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Executive Summary 


The Student-Centered Arts-Learning Environments project (SCALE Project) focused 
on professional development for teachers that enabled them to integrate arts into other cur- 
ricular areas through a model of cross-discipline collaboration. Teachers prepared to lead 
elementary school students through the planning and implementation of an arts-based 
cross-curricular project, and to attend a theme-related concert performed by the West Vir- 
ginia Symphony Orchestra (WVSO). 


During late spring 2012, staff from the WVSO and West Virginia Department of Edu- 
cation (WVDE) Office of Instruction recruited 16 schools in high poverty areas to participate 
in the SCALE Project during the following fall. Each school sent a team (including a regular 
education teacher, arts teacher, and a school administrator) to two face-to-face training ses- 
sions (in June and October 2012) and a webinar (August 2012). During the subsequent 
weeks, teams implemented their projects and prepared their students to participate in a per- 
formance (November 2012) of Maestro Grant Cooper's Boyz in the Wood, a contemporary 
children’s concert based on the story of Little Red Riding Hood. 


The overall goals of the SCALE Project included increases in (a) peer collaboration 
among teachers for interdisciplinary instructional planning and implementation of arts inte- 
gration; (b) teacher knowledge of arts integration and lesson design; and (c) student en- 
gagement in learning and in music and arts. The WVDE Office of Research with support 
from the Office of Early Learning, Office of Instruction, and WVSO undertook a collabora- 
tive program evaluation to measure progress toward those goals. The evaluation addressed 
five core evaluation questions (EQs) as described in the results section below. 


Methods 
The primary sources of data were the following: 


e The SCALE Project Professional Development Evaluation Survey was deployed to 
SCALE Project team members onsite at the end of their final training session in Oc- 
tober 2012. We received 28 responses from the 32 participants, for an 87.5% re- 
sponse rate. 


e The SCALE Project Implementation Rubric and Checklist was used from October 
2012 through early April 2012 by WVDE and WVSO staff/consultants to record ob- 
servations for 15 of the 16 schools in the project, representing 93.8%. 


e The Project SCALE School Survey (Pre Survey) was deployed to the entire school fac- 
ulty in SCALE schools early in the fall before the implementation of SCALE projects 
and again the following February after projects had ended. Only respondents of the 
Pre Survey participated in the Post Survey. For the Pre Survey we received 164 re- 
sponses from 371 teachers contacted (44%). Of those 164 respondents to the Pre Sur- 
vey, we received 111 Post Survey responses (68%). 
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Descriptive statistics and qualitative data analysis were used to summarize data re- 
lated to EQi, including measures of central tendency and dispersion (e.g., mean/standard 
deviation) as well as percentages of respondents indicating specific perceptions of the 
SCALE program. Likewise for EQ2, we used descriptive statistics to summarize implementa- 
tion fidelity data, describing the number and proportion of schools that met adequate fideli- 
ty on the core indicators noted above. To address EQ3, EQ4, and EQ5, we conducted a series 
of paired t tests whereby we ascertained if changes from pre- to postintervention were statis- 
tically significant and in the predicted direction, and we used Hedge’s g test to determine if 
changes observed were substantively important. 


Results 


Evaluation Question 1. What was the quality of the training and technical assistance that 
was provided to SCALE Project schools? 


Professional development offered to participants in the SCALE Project was well at- 
tended and well received. The overall ratings were remarkably high, with participants 
strongly agreeing about the high quality of the training and trainers on multiple measures, 
and approaching strong agreement about the high quality of the materials provided. The 
comments from participants also strongly praised the program, materials, and trainers. A 
few indicated they did not feel prepared to implement the project in their schools—that is, 
they still had lots of questions—after the summer professional development, but when they 
had completed all three training events, they indicated they were looking forward to using 
what they had learned back in their schools and classrooms. Overall, based on participant 
perceptions, SCALE Project team leaders were well trained to facilitate the program in their 
schools. 


Evaluation Question 2. How well did the SCALE Project schools implement the central 
components of the program? 


Rubric and benchmark data provided by West Virginia Symphony Orchestra (WVSO) 
and West Virginia Department of Education (WVDE) personnel indicated that in most 
schools, the SCALE Project was well implemented (80%). As a group, rubric data indicated 
that schools saw the greatest level of implementation relative to student engagement in the 
arts and the lowest level of implementation relative to improving lesson design. Other data 
showed the following: (a) there was agreement or strong agreement that student engage- 
ment was high on all measures; (b) in the large majority of schools, at least six of 10 role 
groups were involved in the project, with Title I teachers, physical education teachers, par- 
ents, and community members the least often involved; (c) the arts were well integrated 
into the SCALE school projects; and (d) the large majority of schools implemented eight of 
nine major SCALE Project components. 


Implementation was far from even, however. Four schools had implementation rates 
above 90% (high-implementation schools), while four had implementation rates below 75%, 
ranging down to 51% (low-implementation schools). For those that had lower levels of im- 
plementation, several issues seemed to be the most common: (a) forming a team, (b) arrang- 
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ing for it to meet regularly; (c) limited arts integration professional development for staff; 
(d) limited involvement of content areas other than the arts, or (e) involvement of only 
some, not all, of their classrooms. 


The unevenness in implementation did not seem related to previous experience with 
the program (i.e., all four low-implementation schools had previous experience with the 
SCALE Project, as did three of the four high-implementation schools), nor with schools’ in- 
volvement in the School Improvement Grant (SIG) program (half of the low implementation 
and half of the high implementation schools were SIG schools). Although it is not possible to 
say with certainty, successful implementation seemed related to the school’s commitment to 
the project—which included establishing an active, multidisciplinary team, strong collabora- 
tion, and involvement of the whole school. 


Evaluation Question 3. To what extent did the SCALE Project result in increased student 
engagement in music and the arts and in other content areas? 


SCALE school teacher surveys conducted before and after implementation of the pro- 
ject contained three subscales measuring educators’ perceptions of student affective, behav- 
ioral, and cognitive engagement in the arts and other content areas. All three subscales 
showed small improvements, although only one, behavioral engagement, approached statis- 
tical significance. When looking at individual survey items that made up the subscales, two 
interesting findings emerged—teachers reported small positive effects for students staying 
on task and for student motivation. 


When comparing schools with prior experience in SCALE with those new to the pro- 
ject, there were small but substantively important positive changes in both behavioral and 
cognitive engagement for schools new to SCALE. Looking at individual items, we found two 
of the strongest effects we detected in the study: Teachers in new-implementation schools 
reported higher levels of students staying on task at Post Survey than at Pre Survey and 
higher levels of students believing they were learning in their classes. For prior- 
implementation schools we found there were slightly lower perceptions of the extent to 
which students were excited about their schoolwork at Post Survey. 


As noted in the discussion of EQ2, there were large differences among the schools in 
their levels of implementation, so we looked at what bearing, if any, implementation fidelity 
had on changes between Pre and Post Survey. With respect to overall subscale scores, we 
found that educators in high-fidelity schools reported both higher overall behavioral and 
cognitive engagement among students at Post Survey than at Pre Survey. When we looked a 
little deeper at individual student engagement items, educators in high-fidelity schools at 
Post Survey reported (a) students stayed on task more, (b) preferred more challenging as- 
signments, and (c) followed instructions better. 


Evaluation Question 4. To what extent did the SCALE Project impact culture and climate in 
participating schools? 


Our analysis of pre- and posttest responses to school culture items on the school sur- 
vey revealed no statistically significant or substantively important changes from pre- to post- 
test data collections on any of the four subscales. These findings held when we disaggregated 
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the responses by new or prior implementation and by high- and low-implementation fideli- 
ty. This is not surprising for such a brief intervention such as the SCALE Project. For an in- 
tervention that lasted only a few months, it would be unusual to see significant changes. 


Evaluation Question 5. To what extent did the SCALE Project result in improved lesson de- 
sign and/or instructional delivery among participating schools? 


We constructed three subscales in the Project SCALE School Survey (Pre and Post 
Survey versions) for lesson design and instructional delivery to measure (a) integration of 
arts content into instruction, (b) collaboration among faculty role groups and community 
members in lesson design, and (c) collaboration among school-based teams in lesson plan- 
ning. We found no statistically significant or substantively important changes at the subscale 
level for the whole group, nor for schools disaggregated by experience with the SCALE Pro- 
ject (new versus prior), nor for schools disaggregated by level of fidelity of implementation 
(high versus low). We also disaggregated responses by content area (arts versus other con- 
tent areas), and again did not find changes to report. When looking at individual items, how- 
ever, there were several interesting—albeit small effects at Post Survey: 


e Teachers from new-implementation schools reported more collaboration with com- 
munity members in the development of their lessons, and more integration of dance/ 
movement strategies into their own instruction. 


e Non-arts elementary teachers reported more integration of dance/movement strate- 
gies into their own instruction, and more collaboration with arts teachers. 


e Teachers from high-implementation-fidelity schools reported less integration of 
creative writing instructional strategies, and more integration of dance instructional 
strategies. 


Limitations of the Study 


Several limitations of this study bear mentioning. First, all data are self-reported and 
thus subject to various threats to validity, such as social desirability bias (when respondents 
provide overly positive responses to a survey or questionnaire due to their desire to be 
viewed favorably) or nonresponse bias (when respondents who elect not to participate in a 
survey differ in a meaningful way with those who do). We encountered one technical difficul- 
ty, when we unintentionally excluded an item in one of the subscales drawn from the POSC 
instrument. To compensate for the missing item, we applied a multiplier. There is a small 
possibility that the validity of the measure was affected, but there was little or no change in 
these measures, so this error did not impact our findings in any meaningful way. Another 
limitation of our study involves our inability to draw school-level conclusions regarding the 
impact of the various projects implemented by schools, because we did not receive a large 
number of completed surveys from many schools to make such disaggregation possible. 
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Recommendations 
We make the following recommendations based upon our results: 


To the extent possible, we recommend continuing this project. Educators appear to 
perceive positive benefits of the program for their students especially in the areas of 
cognitive and behavioral engagement. These are important outcomes that could lead 
to improved student achievement if sustained. 


Efforts should be made to sustain initial excitement so that prior implementation 
schools can continue to realize benefits. In this study, we found that new implemen- 
tation schools realized more positive outcomes than prior-implementation schools. 
The excitement factor in new schools could have contributed to these findings. 


Encourage and support full implementation of all components of the program. We 
found that higher-than-average implementation fidelity schools experienced more 
positive outcomes than those schools that did not implement many components with 
fidelity. We found no significant changes for lower-than-average implementation 
schools. That is, while failure to implement the program as intended is not necessari- 
ly associated with negative outcomes, it does potentially maintain the status quo. 
Program staff should use these results as a catalyst for participating schools, to illus- 
trate that a school’s level of commitment can make or break the project. 


Ensure that schools participating in the project build in sufficient common planning 
time to support the necessary collaboration. This time is essential to ensure the 
school’s project is implemented with fidelity and achieves the intended school-wide 
outcomes. Administrator support is critical in this regard and should be discussed 
early on in the project. 


Develop strategies to ensure that once the school project concludes, the faculty does 
not return to business as usual. One strategy may be finding ways to sustain the 
momentum of the project—that is to continue on with other collaborative projects 
that integrate various content areas, including the arts. Another strategy may involve 
addressing beliefs that time spent on a project such as SCALE is time taken away 
from improving test scores in mathematics and reading/language arts. Helping edu- 
cators understand the strong connections between high student cognitive and behav- 
ior engagement—as seen in many of the SCALE schools this year—and high student 
achievement could help reduce the anxiety felt about making the sorts of changes in 
lesson planning and instruction that the SCALE project encouraged. 
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Introduction 


The Student-Centered Arts-Learning Environments project (SCALE Project) was de- 
veloped by the West Virginia Symphony Orchestra (WVSO) and piloted in collaboration with 
Regional Education Service Agency (RESA) 3 in four schools during the fall of 2009. In each 
subsequent year, the SCALE Project has increased the number of participating schools, al- 
ways targeting schools in high poverty areas. The SCALE Project focuses on professional de- 
velopment for teachers to enable them to integrate arts into other curricular areas through a 
model of cross-discipline collaboration. By participating in the project, teachers prepare to 
lead elementary school students through the planning and implementation of an arts-based 
cross-curricular project, and to attend a theme-related concert performed by the WVSO. 


During late spring 2012, staff from the WVSO and West Virginia Department of Edu- 
cation (WVDE) Office of Instruction recruited 16 schools to participate in the SCALE Project 
during the following fall. Each school sent a team to an initial training session held on June 
19, 2012, at the Clay Center (home of the WVSO) in Charleston, West Virginia. The teams 
included a regular education teacher, arts teacher, and a school administrator. One of the 
team members was designated as the school liaison, and served as the main point of contact 
at the school throughout the project. At the initial training team members learned about the 
requirements of the project, approaches to cross-disciplinary collaborative lesson planning 
to support an arts-related school project, and their role in the research component of the 
project. Regarding the latter, participants were given a one-page description of the SCALE 
Project Review Criteria (see Appendix A), to inform them about the criteria that would be 
used in assessing the implementation of the project in their schools. 


Initial training was completed during a follow-up webinar, held August 14, 2012. Par- 
ticipants were given the opportunity to ask questions and receive answers as they prepared 
to deliver training to their peers in their own schools, scheduled to take place on or before 
September 15, 2012. Participants also received a data collection schedule (Appendix A). 


School teams gathered one last time on October 9, 2012, to review and critique each 
other’s school project plans. During the subsequent weeks, teams implemented their plans 
and prepared their students to participate in a performance of Maestro Grant Cooper's Boyz 
in the Wood, a contemporary children’s concert based on the story of Little Red Riding 
Hood. The project culminated with students’ and teachers’ participation in one of three con- 
cert performances (two in Charleston and one in Morgantown) in November. Participation 
was free to all SCALE Project schools, supported by a grant from the Claude Worthington 
Benedum Foundation. 


Goals of the Evaluation 


The overall goals of the SCALE Project included increases in (a) peer collaboration 
among teachers for interdisciplinary instructional planning and implementation of arts inte- 
gration; (b) teacher knowledge of arts integration and lesson design; and (c) student en- 
gagement in learning and in music and arts. The WVDE undertook a collaborative program 


Introduction 


evaluation of the SCALE project in the 2012-2013 school year to measure progress toward 
those goals. The evaluation was conducted by the WVDE Office of Research, with support 
from the Office of Early Learning and the Office of Instruction, and the WVSO. The first step 
in the evaluation was to develop a logic model for the project (see Appendix B). Based on 
discussions and the logic model, the SCALE program evaluation addressed five core evalua- 


tion questions (EQs): 


EQu. 
EQ2. 
EQ3. 
EQ4. 


EQ5. 


What was the quality of the training and technical assistance that 
was provided to SCALE Project schools? 

How well did the SCALE Project schools implement the central 
components of the program? 

To what extent did the SCALE Project result in increased student 
engagement in music and the arts and in other content areas? 

To what extent did the SCALE Project impact culture and climate in 
participating schools? 

To what extent did the SCALE Project result in improved lesson design 
and/or instructional delivery among participating schools? 
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Participant Characteristics 


Participants in the project were primarily teachers and students from 16 schools 
serving large proportions of students from low-income families. Details about the schools, 
including their Title I status, No Child Left Behind accountability status, their involvement 
(or not) in the federally funded School Improvement Grant program, and previous experi- 
ence with the SCALE project, are available in the next section or in Table A 1 (page 81). 


We used a variety of methods to address the evaluation questions, depending on the 
nature of each question, as described below. 


Evaluation Question 1 


EQ1. What was the quality of the training and technical assistance that was provided 

to SCALE Project schools? 

To address EQ1 we developed and deployed the SCALE Project Professional Devel- 
opment Evaluation Survey (Appendix C) which included items to measure the quality of the 
initial training, follow-up webinar, and final training provided by the West Virginia Depart- 
ment of Education (WVDE) in collaboration with the West Virginia Symphony Orchestra 
(WVSO). Items assessed the general quality, relevance, and usefulness of the training and 
materials, and the quality of facilitators. 


Evaluation Question 2 


EQ2. How well did the SCALE Project schools implement the central components of 

the program? 

To address EQ2 we developed a SCALE Project Implementation Rubric and Check- 
list in collaboration with the WVDE Office of Instruction and the WVSO. The rubric consists 
of five key indicators, as well as a list of key implementation milestones such as the estab- 
lishment of a SCALE Project team, regular meetings of the SCALE Project team to discuss 
arts integration, and so forth (see Appendix C for the full rubric and checklist). WVSO staff/ 
project consultants and WVDE Office of Instruction/Office of Early Learning staff used the 
rubric to provide information about the level of implementation fidelity of SCALE Project in 
each school. WVDE Office of Research staff analyzed data in the completed rubrics to de- 
termine the extent to which each school completed the core components of the SCALE pro- 
ject. The implementation data from the rubrics was also used as a method for disaggregating 
additional data described in the results section. Table 1 provides an overview of the five ma- 
jor implementation indicators. 


Methods 


Table 1. SCALE Project Core Implementation Categories 


Indicator Ideal implementation status 


Student engagementin Asa result of the project, students exhibit a higher level of engagement in the 

learning classroom, not only physically, but through demonstration of cognitive engagement 
with the content and affective engagement in the learning process, driven by 
student inquiry. 

Student engagement in Asa result of the project, students have increased engagement in arts learning, 


the arts both through general instruction in music and the other arts, but also through 
extended arts learning experiences. 

Collaboration for arts As a result of the project, classroom teachers use standards-based arts integration 

integration and other as a regular teaching strategy and, when applicable, collaborate with school arts 


interdisciplinary learning teachers to design instruction. 

Improvements inlesson Asa result of the project, teachers personalize learning to a greater degree; use 

design student inquiry as a primary instructional method; promote greater student 
collaboration; and demonstrate a deeper understanding of standards-based 
instructional design, including learning experiences aligned to standards, teaching 
to mastery, designing for engagement, and acceptable evidence of learning. 

Improvements inschool Asa result of the project, a greater number of faculty are working together; both 

culture students and teachers have taken a greater responsibility for learning; the physical 
environment of the school has become more inviting; and students are allowed 
greater latitude in decision making. 


Evaluation Questions 3, 4 and 5 


EQ3. To what extent did the SCALE Project result in increased student engagement 
in music and the arts and in other content areas? 


EQ4. To what extent did the SCALE Project impact culture and climate in partici- 
pating schools? 


EQ5. To what extent did the SCALE Project result in improved lesson design and/or 

instructional delivery among participating schools? 

To address EQ3, EQ4, and EQ5 we developed two questionnaires that served as pre- 
and postintervention survey instruments (See Appendix C, page 43). All staff in SCALE 
schools were invited to complete the Pre Survey, but only respondents to the Pre Survey 
were invited to respond to the Post Survey, to assure that we had a matched sample of the 
same individuals pre- and postintervention. Because the survey was fairly lengthy and the 
Pre Survey respondents already had the experience of filling it out once, we offered an incen- 
tive for responding to the Post Survey. The WVSO offered three Best Buy gift certificates 
(worth $250, $100, and $50) and vouchers for concert tickets for three randomly selected 
Post Survey respondents. 


The survey questionnaires differed only in the reference frame respondents were 
asked to reflect upon as they answered questions (“last school year” or “this school year”). 
Both questionnaires consisted of three main sections: (1) student engagement in learning 
and the arts, (2) collaboration for arts integration and improved lesson design, and (3) 
school culture/climate. Items for the first two sections were developed by the WVDE, based 
upon adaptations of items included in the Student Engagement in Schools Questionnaire 
(Hart, Stewart, & Jimerson, 2011), the National Center for School Engagement’s (NCSE) 
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Student Survey (National Center for School Engagement, 2004), and the Post-Survey for 
Teachers involved in the RESA 3/Symphony Arts Project. Items for the third section (school 
culture/climate) were taken verbatim from the validated and nationally normed Perceptions 
of School Culture survey (Cowley, Voelkel, Finch, & Meehan, 2006). 


Data Analysis 


Descriptive statistics and qualitative data analysis were used to summarize data re- 
lated to EQi, including measures of central tendency and dispersion (e.g., mean/standard 
deviation) as well as percentages of respondents indicating specific perceptions of the 
SCALE program. Likewise for EQ2, we used descriptive statistics to summarize implementa- 
tion fidelity data, describing the number and proportion of schools that met adequate fideli- 
ty on the core indicators noted above. To address EQ3, EQ4, and EQ5, we conducted a series 
of paired t tests whereby we ascertained if changes from pre- to postintervention were statis- 
tically significant and in the predicted direction, and we used Hedge’s g test to determine if 
changes observed were substantively important. Additional details about the methods of 
analysis we used are presented along with the results in the next section. 
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The primary sources of data in this section were the following: 


e The SCALE Project Professional Development Evaluation Survey was deployed to 
SCALE Project team members onsite at the end of their final training session, on 
October 9, 2012 (Appendix C, page 43). We received 28 responses from the 32 partic- 
ipants, for an 87.5% response rate. 


e The SCALE Project Implementation Rubric and Checklist (Appendix C, page 48) was 
deployed on October 18th; data collection ended on April 9, 2013. West Virginia De- 
partment of Education (WVDE) and West Virginia Symphony Orchestra (WVSO) 
staff/consultants submitted records for 15 of the 16 schools in the project, represent- 
ing 93.8%. 


e The Project SCALE School Survey (Pre Survey) was deployed to the entire school fac- 
ulty in SCALE schools early in the fall (October 2—October 17, 2012; see survey in- 
strument and e-mail invitation messages, Appendix C, page 54). After the SCALE 
project concluded (February 7—March 7, 2013), we deployed the very similar Project 
SCALE Post-Intervention School Survey (Post Survey), inviting only respondents of 
the Pre Survey to respond to the Post Survey (see survey instrument and e-mail invi- 
tation messages, Appendix C, page 68). For the Pre Survey we received 164 responses 
from 371 teachers contacted (44%). Of those 164 respondents to the Pre Survey, we 
received 111 Post Survey responses (68%). 


Baseline Data About Participants in the Project 


As noted earlier, the target group for the SCALE Project was elementary schools serv- 
ing high poverty areas. Table A 1 (in Appendix D, page 81) shows that 15 of the 16 schools 
recruited were eligible for school-wide Title I funding; additionally, half of the recruited 
schools had failed to make adequate yearly progress (AYP) for at least 3 consecutive years 
and were identified as being in need of improvement, corrective action, or restructuring. 
Five of the schools were federal School Improvement Grant recipients and were, therefore, 
subject to interventions in addition to their voluntary participation in the SCALE Project. 
Half of the schools had been involved in the SCALE project previously, although not neces- 
sarily in the immediate previous year. It should also be noted that one school, Poca Elemen- 
tary, was designated as a 2012 Title I Distinguished School by the WVDE Office of Federal 
Programs. The schools were in 13 county districts located across the state. 


Evaluation Question 1 


What was the quality of the training and technical assistance that was provided to 
SCALE Project schools? 


Results 


SCALE Project _ staff 
planned for and conducted three 
professional development  ses- 
sions. The two face-to-face ses- 
sions were generally well 
attended, with participants from 
14 of the 16 schools in the project 
at each meeting. Additionally, 
approximately 18 participants 
signed onto the August webinar. 
In addition to the professional 
development sessions, staff or 
consultants from the WVSO or 
the WVDE Office of Instruction 
or Office of Early Learning visit- 
ed each of the schools to provide 
onsite technical assistance. Of 
the 34 attendees at the final Oc- 
tober 2012 session, 28 (82%) 
filled out survey questionnaires. 
Of those respondents, only a 
third (33%) indicated they had 
been in their current positions 
for more than five years, while 
two thirds (67%) had been work- 
ing in the education field for 
more than 5 years. This indicates 
that SCALE Project team leaders 


Table 2. SCALE Team Member Participation in Professional 
Development and Survey* 

Attended Attended Responded 
June PD October PD to PD survey 
(n) (n) (n) 
Total 40 34 28 
Ansted Elementary 3 1 0 
Brookview Elementary 2 0 0 
Burch Elementary 4 3 3 
Culloden Elementary 5 0 (6) 
Dingess Elementary 3 2 1 
Doddridge Elementary 2 2 2 
Geary Elementary d 1 4 
Guyandotte Elementary 4 3 3 
Lizemore Elementary 3 3 2 
Poca Elementary 2 1 1 
Reedy Elementary 0 3 2 
Romney Elementary 3 4 3 
Smoot Elementary 2 2 2 
Spencer Elementary 2 2 2 
Watts Elementary 1 1 1 
Weimer Elementary 0 3 2 
Non-SCALE schools** 3 3 3 


*Participant data were not collected for the approximately 18 


participants in the August webinar. 


**Non-SCALE schools were invited to send arts teachers to the 
trainings and some classrooms to the concerts. Teachers from 
these schools received materials but were not required to form a 


team. 


were experienced teachers who were relative newcomers to their schools. Just over a third 
(38%) of the respondents were arts teachers; the rest were regular elementary education or 


Title I teachers. 


The survey was divided into three sections: (a) overall quality, relevance, and useful- 
ness of the training; (b) adequacy of the materials and resources provided; and (c) adequacy 


of the facilitator(s). 


Quality of training 


The overall assessment of the training’s adherence to research-based practices for 


professional development was very high (Table 3). The mean composite score for seven 
items combined was 4.6 on a 5-point Likert-type scale, including 1 (strongly disagree), 2 
(disagree), 3 (neutral), 4 (agree), and 5 (strongly agree). The only item that did not fall with- 
in the strongly agree range (4.3) was “Training objectives were clearly stated before sessions 
began.” 


Results for other items, rated on rubric-type scales, can be found in Table A 2 
(Appendix D, page 82). Highlights of those ratings include the following: 
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e The majority of participants considered the training “a good start” and look forward 
to using what they learned. 


e Participants were evenly divided in indicating they “already practice/apply” or they 
“look forward to practicing/applying” the knowledge and skills they learned. 


e Just over 40 percent indicated the professional development was only “somewhat 
aligned” or “not aligned” with their school’s/program’s goals for instructional im- 
provement. 


e Nearly two thirds thought the professional development was more useful than other 
professional development they had participated in. 


Clearly participants in the training thought highly of it as a professional development 
experience. 


Table 3. Participant Ratings for Quality of Training, Material/Resources, and Trainers 


Number Minimum Maximum Mean SD 


Quality of training (adherence to research-based practices for high quality professional development) 


The training was high quality. 27 4.00 5.00 4.63 492 
The training was relevant. 28 4.00 5.00 4.71 .460 
The training was well organized. 27 3.00 5.00 4.52 .700 
The training was specific and content- 28 4.00 5.00 4.61 .497 
focused. 
The training was hands-on and included 28 3.00 5.00 4.57 573 
active learning opportunities. 
Training objectives were clearly stated before 28 3.00 5.00 4.36 .870 
sessions began. 
Training sessions began and ended in a timely 28 4.00 5.00 4.57 504 
fashion. 

Quality of materials and resources 
Adequate amounts of training materials/ 27 3.00 5.00 4.37 .629 
resources were provided. 
Materials/resources were relevant to my 26 3.00 5.00 4.42 .643 
work. 
The materials/resources provided were of 27 3.00 5.00 4.59 572 


high quality (i.e., based on recent research 
and evidence-based). 
The materials/resources provided were 27 3.00 5.00 4.44 .698 
useful to my work. 
Quality of trainers 


Trainer(s) were knowledgeable about the 27 4.00 5.00 4.78 424 
topic. 

Trainer(s) were well organized. 27 3.00 5.00 4.59 .694 
Trainer(s) presented the material clearly and 26 3.00 5.00 4.65 .629 
effectively. 

Trainer(s) facilitated discussions well. 27 4.00 5.00 4.78 424 
Trainer(s) answered questions raised during 27 4.00 5.00 4.70 465 


sessions adequately. 
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Quality of materials 


Using the same 5-point Likert-type scale, participants also rated the materials and 
resources they received very highly (composite mean score, 4.5), with participants most 
strongly agreeing (4.6) that “materials/resources provided were of high quality (i.e., based 
on recent research and evidence-based).” The most tepid agreement was for “Adequate 
amounts of training materials/resources were provided,” but even this item scored only 
slightly below strongly agree, with a mean score of 4.4 (Table 3). 


Quality of trainers 


The highest ratings on the 5-point Likert-type items were assigned to the trainers. 
The overall mean for these five items was 4.7, and none of the items fell below the range for 
strongly agree. The highest of these very high agreement scores (4.8) went to “Trainer(s) 
were knowledgeable about the topic” and “Trainer(s) facilitated discussions well” (Table 3). 


Responses to open-ended questions 


Participants were asked four open-ended questions, one each about the training 
overall, materials/resources, and trainers; and one that allowed other comments. With re- 
gard to the training overall, most were expressions of enthusiasm for the training and the 
program, such as the following: 


This training should be mandatory. .. 


These PD’s have been more inspiring than any others I have attended in the last 10 
years. I have even used some of the inquiry techniques to jump start some of my per- 
sonal creative work—with encouraging results. 


This program has made me expand my perception of how to teach CSO's. I will now 

look at where I want students to go and then plan what has to be done to get them 

there. 
One respondent explained the slightly lower rating for the training’s alignment with school/ 
program goals, indicating that the goals need to catch up with the training. 

The PD was NOT ALIGNED with school's/program's goals for instructional im- 

provement.—Only because they aren't there yet. 
Two participants indicated that the questions they were left with at the end of the June train- 
ing had been answered during this training, so they felt prepared to move forward with the 
program. 


Of the 10 open-ended comments about materials, eight expressed enthusiasm and 
gratitude. Examples included the following: 


As a participant, our school has received or has been given access to many re- 
sources/supplies. Awesome! 


The basket of educational "goodies" was wonderful! The teachers have used all of the 
resources in them. Thank You! 


The resources were great. My school doesn't have an art or music teacher so having 
any resources helped greatly. 


Three of the respondents praised the knowledge and passion of the trainers, such as 
the following: 
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Excellent Trainings - Very entertaining trainers; knowledgeable; passionate about 

their skills/curriculum, grasping onto change and just very enthusiastic!! 

Two respondents offered constructive criticism, both with regard to allowing more “wait 
time” during discussions, as illustrated in this comment: 

Please be careful with "wait time"—when we were asked to do an activity or answer a 

question, there was not enough time to think before we moved on, or the presenter 

continued to talk during that time. 

The final item in the survey offered respondents the opportunity to provide other 
comments; fourteen participants responded. Of those, six were additional expressions of en- 
thusiasm for various aspects of the project, including the opportunity for students to go to 
the symphony concert and to incorporate arts in their classrooms. 


I really enjoyed listening to Grant Cooper speak and discussing what we are doing 
with the other schools. 


This is our third year to participate. Every year just keeps getting better! 


There were three complaints about shortages of food during the October training. 


Evaluation Question 2 


How well did the SCALE Project schools implement the central components of the 

program? 

WVSO and WVDE staff, who served as technical assistance providers during the 
course of the project, responded to the SCALE Project Implementation Rubric and Check- 
list, collectively submitting one response for each school. The exception was Smoot Elemen- 
tary School. Multiple unsuccessful attempts were made over the course of three months to 
schedule a visit at this school following the conclusion of the project. WVDE staffs inability 
to make those arrangements may have been due to changes in the school team associated 
with the SCALE Project. Consequently, the following findings are based on 15 of the 16 
schools in the project. 


The rubric included in the instrument (items 6, 8, 10, 12, and 14) described four lev- 
els of implementation (see Table 1 for rubric details). Outcomes for each of the schools ap- 
pear in Table 4. The levels in the rubrics are each qualitatively described, but they 
correspond to 1 (no evidence of implementation), 2 (low level of implementation), 3 (moder- 
ate level of implementation), or 4 (high level of implementation). As a group, schools saw the 
greatest level of implementation relative to student engagement in the arts (mean score of 
3.2) and the lowest level of implementation relative to improving lesson design (2.8). There 
was great variation in the level of implementation realized among the schools, with three 
schools (Geary, Watts, and Brookview) achieving only the lowest level of implementation, 
and three schools (Poca, Lizemore, and Weimer) achieving a high level implementation, with 
the remaining nine schools achieving moderate levels on the five components. 
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In addition to the implementation rubric, item 5 on the instrument asked about the 
ability of students to engage in various behaviors promoted by the project. These questions 
asked the WVSO/WVDE respondents to indicate their level of agreement on the following 
scale: 1 (strongly disagree), 2 (disagree), 3 (agree), or 4 (strongly agree). Table 4 shows that, 
overall, respondents agreed that through the project students were able to 


e Connect content among multiple disciplines; 

e Represent complex ideas; 

e Share created products with others; 

e Participate fully as individuals; 

e Collaborate meaningfully with each other; 

e Participate in higher-order learning conversations; 
e Engage meaningfully in the arts; and 


e Work collaboratively with the WVSO 


Table 4. Level of Implementation by Central Component, by School 


Student Student Collaboration Improvements Improvements 

engagement engagement for arts in lesson in school 
School in learning in the arts integration design culture Mean Rating 
Overall 3.1 3.2 3.0 2.8 2.9 3.0 
Poca Elementary 4 4 4 4 4 4.0 
Lizemore Elementary 4 4 4 3 4 3.8 
Weimer Elementary 3 4 4 4 4 3.8 
Burch Elementary 4 4 4 3 2 3.4 
Guyandotte Elementary 3 4 3 3 3 3.2 
Spencer Elementary 4 4 3 2 3 3.2 
Doddridge Elementary 4 3 3 3 3 3.2 
Ansted Elementary 3 3 3 3 3 3.0 
Dingess Elementary 3 3 3 3 3 3.0 
Romney Elementary 3 3 3 3 3 3.0 
Culloden Elementary 3 3 4 3 2 3.0 
Reedy Elementary 2 2 2 3 4 2.6 
Geary Elementary 3 3 2 2 2 2.4 
Watts Elementary 2 2 2 2 2 2.0 
Brookview Elementary 2 2 1 1 1 1.4 


Once again, however, there was tremendous variation across the SCALE Project 
schools. For seven schools there was strong agreement that all or nearly all of these behav- 
iors were enabled, while for eight others there was general agreement, with mean scores 
ranging from 2.8 to 3.5 (see Table A 3, Appendix D, page 83). 
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Looking across schools, 
the role groups least included 
were community members and 
parents, with only 20.0% and 
38.5% of the schools including 


Figure 1. Mean Scores for Student Behaviors Engaged in 
Through SCALE Project Across SCALE Project 
Schools 
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Figure 2. Percentage of SCALE Project Schools That Involved | all or all but one of the 10 role 

Various Role Groups in Their Project groups—were Weimer, Poca, 
Burch, and Ansted. Least inclu- 
sive were Brookview, Reedy, and 
Watts, each including only three 
or four of the 10 roles groups 
(Table A 4, Appendix D, page 84). 
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training from the WVSO and 
Figure 4. Percentage of SCALE Project Schools That WVDE.! One of the last items 
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than half of the components, including Reedy, Brookview, and Watts Elementary Schools 
(see Table A 6, Appendix D, page 85). 
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Our last analysis of project implementation involved developing a single, composite 
implementation score for each school. The purpose of the score was to test what bearing var- 
ious levels of implementation had on measures discussed in the remaining evaluation ques- 
tions (see below). To develop the score involved several steps, as outlined next. 


We had two types of items in the Project Implementation Rubric and Checklist. The 
first type was based on a 4-point, Likert-type scale. The second type involved a checklist, 
where respondents simply checked yes or no. To develop a composite score, we treated each 
of these two types using different methods. 


We combined the Likert-type items (5, 6, 8, 10, 12, and 14) and computed mean 
scores for each school; we then converted those scores to percentages, with a 4.0 mean score 
considered to be a perfect, 100% implementation. Lower scores were converted to percent- 
ages of 4.0, so for example, a mean score of 3.0 was converted to 75% (see columns A and B 
in Table 5). 


Similarly, we combined the checklist items (3, 4, and 16), and assigned a 1 for yes and 
a O for no. For each school, we counted across the items (there were some missing values, 
where the WVSO or WVDE respondent did not have enough information to respond to the 
question), and then divided the sums for each school by the counts, to get percentages of 
yeses (see column C in Table 5). 


1 This level of participation was encouraged but voluntary. One school involved classrooms 
from only one grade level. 
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Lastly we calculated the mean of the two percents to get the aggregate mean percent 
(see column D in Table 5). The aggregate median for the whole project was 85.9%, with eight 
schools achieving 85.9% implementation or above (the highest being Poca Elementary at 
98%); and eight schools scoring at or below that level (the lowest level of implementation 
taking place at Watts Elementary, at 51%). The overall mean was 81% implementation. 


Table 5. Composite Implementation Scores by School 
Column A Column B Column C Column D 
Percentages of 
Likert item means checklist items 

converted to markedas Aggregate mean 
School Likert item means percentages of 4.0 implemented (yes) percent 
All schools 3.32 82.9 78.6 80.8 
Poca Elementary 4.00 100.0 96.0 98.0 
Weimer Elementary 3.85 96.2 96.0 96.1 
Spencer Elementary 3.69 92.3 92.0 92.2 
Doddridge Elementary 3.69 92.3 90.9 91.6 
Guyandotte Elementary 3.69 92.3 84.0 88.2 
Burch Elementary 3.69 92.3 83.3 87.8 
Culloden Elementary 3.38 84.6 88.0 86.3 
Dingess Elementary 3.54 88.5 83.3 85.9 
Romney Elementary 3.31 82.7 84.0 83.3 
Lizemore Elementary 3.62 90.4 76.0 83.2 
Ansted Elementary 2.92 73.1 91.7 82.4 
Geary Elementary 2.69 67.3 80.0 73.7 
Reedy Elementary 2.77 69.2 48.0 58.6 
Brookview Elementary 2.46 61.5 45.8 53.7 
Watts Elementary 2.46 61.5 40.0 50.8 


Note: the median of aggregate mean percent implemented = 85.9 


Qualitative data 


WVDE and WVSO technical assistance providers who responded to the implementa- 
tion rubric were asked to describe the school projects; those descriptions appear in 
(Appendix D, page 87). Highlights from three school projects include the following: 


e Poca Elementary—The project was announced at a whole school assembly, inviting 
students to be as creative as they liked. They were provided a number of resources to 
do so. The primary grades . . . took Little Red around the world to every continent 
and discussed the weather and transportation there . . . Other students compared 
and contrasted Red with other fairy tale characters. Third grade students created a 
puppet theater . . . [Students] created a grocery list for Red's basket, budgeted the 
amounts, and visited a grocery to purchase the items. Fifth grade put Little Red on 
trial, each student choosing a part, . . . They also choreographed and performed a 
dance for the play. One class compared the skeleton of the wolf to that of a dog and 
studied the habitat of the wolf... 
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e Weimer Elementary—...Content areas included mathematics, language arts, history 
(an election for President of the Woods), art, music, and performing arts. Each 
classroom contributed to a school mural depicting the Little Red Riding Hood story 
and students from each classroom participated in a culminating celebration, 
performed for parents and guests... 


e Dingess Elementary—. .. Fourth graders created a math forest in a vacant classroom 
which was left up for a month after its creation ... When Maestro Cooper visited in 
December, fourth graders guided Maestro Cooper through the math forest, taking 
pride in describing all the measuring they had to do to create the forest. 


WVDE and WVSO rubric responders were also asked to provide evidence to support 
ratings they assigned to different parts of the rubric. To support high ratings for student en- 
gagement, the following examples of evidence were provided: 


Students were given a choice of activities. There were no wrong answers and no limits 
put on students. 


Writing of second graders was imaginative, thorough, posted all over the walls! First 
graders had been collecting bottle caps which they turned into a collaborative art 
project. 


During WVSO visit, kindergarten students wanted to show their video project which 
they had performed for other classes. Students in older grades shared not only their 
produced projects, but easily answered questions about the processes behind what 
and how they had completed them. Older students chose to adopt a wolf and carry 
the project into the future. 


Maestro Cooper visited with students at the end of January 2013. Students in second 
grade still could recite their audience participation rap. Students asked great 
questions and listened intently to the Maestro. Artifacts submitted by the school in 
November were very student-driven. 


On the other hand, things did not go quite so well at other schools, as evidenced in these 
comments: 

Though there was a great deal of interest in the project in the school, the participants 

had minimal professional development to assist with arts integration. There was a 


good deal of activity, but most of it did not exhibit higher order thinking skills and 
was very teacher directed. 


The staff members worked really hard. Most of the evidence illustrated that students 
occasionally engaged in deeper thinking, but not always. Based on our interactions 
with the staff and observations at the school, the teachers made every effort, but need 
help in how to offer and support higher-order thinking. 


The activities were a series of lower level events connected by the story theme. Rich 
teaching to standards was not evident. 


Evidence of student engagement in the arts at high implementation schools included the 
following examples: 


Artifacts and activities collaboratively created by the students such as an election 
campaign, rap, new story of Red, and understanding of stranger danger. 


Every class was involved (music, visual arts, PE) Energy Express style! 
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Some of the challenges to increasing student engagement in the arts appeared to be related 
to staffing for art and music—or making staff available—as was evident in these observa- 
tions: 


Music and art taught by the same teacher who did not always have the benefit of 
being able to collaborate with each grade level classroom teacher. Projects in arts 
classes were less student driven/derived. Limited time in arts classes also a challenge. 


The school did not directly involve the music teacher and there is no art teacher at the 
school. There was some interest in the arts shown, but most of the focus was on the 
Little Red Riding Hood story itself. 


Regarding the measure, collaboration for arts integration, staff time and old habits 
seemed to be obstacles both during the project and after the project ended, even among 
some of the high-implementation schools. 


Major strides were made in integrating special education specialists; physical 
education and dance was next effective in integration. A single teacher does both 
visual art and music (major time limits prevent more collaboration in this instance.) 
Daily scheduling seen as an obstacle to more collaboration. 


Ideally [there] would be more collaboration, but common planning time is hardest 
obstacle to overcome. 


Carry-over past the project was an issue. Once the project was complete, teachers 
dropped back into comfort zone and worrying about the test. 


Once the [project was] seen as complete, teachers drifted quickly back to comfortable 
ways of teaching. . . Time to collaborate is the biggest issue since many teachers' 
planning periods happen because the students are in music classroom. 


A major emphasis of the professional development associated with the project was improv- 


ing lesson design. Progress was made at some schools, especially where other training (e.g., 
Common Core) aligned with the approaches encouraged in the SCALE Project. 


Students chose their preferred activities and worked together to create products. 
Teachers emphasized standards-based instruction and DOK levels 1-4 as evidenced 
by student work. 


SIG training in Common Core plus additional work being done with an outside 
consultant with this school made them the ideal pilot school. Many initiatives coming 
together reinforced the list of design and evidence. SCALE was a way to effectively 
put everyone on the same storybook page; removed threat of "NEW" stuff. 


School is also involved through the SIG process in Common Core training. SIG, CC 
and SCALE all come together in some cool ways, especially getting teachers to loosen 
up and think about new ways to approach "same old" subject matter. 


WVDE and WVSO rubric respondents reported that the enthusiasm for new approaches to 
lesson design was not universal, however, or it was short-lived. 


The music teacher especially could talk about improvements in these elements, 
particularly in personalizing learning for students. For most other teachers, the 
project seemed to be an "add on." 


Principal reported that teachers were into collaboration throughout the project, 
although keeping the process going dwindled after the project ended. 


{It was hard] for many of the veteran teachers to break free from traditional 
"teaching" and the comfort level of the scripted text. Carry-over of the process is the 
missing link. 
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Finally, the rubric called for evidence of improvements in school culture. At some 
schools, the SCALE Project was reported to have enhanced an already excellent school cul- 
ture or to have moved the SIG process forward. Most reported school-wide enthusiasm and 
involvement in the project. 

[The] principal could not say enough about how their participation in this project had 

gotten teachers to think outside the walls of their own classroom. He specifically 

mentioned the level of camaraderie in the teachers' lounge was "buzzing" with flow of 


ideas and support for fellow teachers. This translated directly into student enthusi- 
asm for learning throughout the course of the project. 


[The] school transformation specialist commented on positive culture change he had 
witnessed in this school, a part of which could be attributed to SCALE. Individual 
teachers really stepped up to be a part of the program, and he gave the impact high 
marks. 
As with the other parts of the rubric, however, there were a few exceptions, where school cul- 
ture was impacted only modestly, and there was less universal involvement in the project, as 
evidenced in one comment: 


Some faculty members appear to be collaborating more, though the lead teacher indi- 
cated difficulty in communication and getting full participation. 


Evaluation Question 3 


To what extent did the SCALE Project result in increased student engagement in mu- 

sic and the arts and in other content areas? 

To address this question, we analyzed responses to the SCALE Project School Pre and 
Post Surveys. The surveys contained three subscales measuring educators’ perceptions of 
student engagement in SCALE schools. These subscales included 13 items measuring affec- 
tive engagement, nine measuring behavioral engagement, and 13 measuring cognitive en- 
gagement (see items 10-12 in Appendix C, page 43). 

We constructed a compo- 


Table 6. Reliability Estimates for Student Engagement Subscales ‘ 
Y = site measure of each type of en- 


Number of gagement by summing re- 

items on Pre survey Post survey ; 
; , earn as, % respondents’ responses to each 
Engagement dimension subscale __ reliability reliability fecal d dividi b 
Affective Engagement 13 .88 .88 subscale item an ~~ Me DY 
Behavioral Engagement 9 .69 74 the total number of items for 
CognniveaEneseement 13 901 93 each subscale. For each subscale, 


*Cronbach’s Alpha (a) was used as the measure of reliability we first assessed the reliability at 


Pre and Post Survey. Table 6 in- 
dicates that, with the exception of the behavioral engagement subscale on the Pre Survey, all 
scales were highly reliable—exceeding the threshold of .70. Notably, the behavioral engage- 
ment subscale met standards for reliability at Post Survey. These findings indicated that it 
was appropriate to examine changes in the composite measures of student engagement, al- 
beit with some caution for behavioral engagement. We also assessed change on each indi- 
vidual item to provide more useful information to program staff and to address the limited 
reliability of the behavioral engagement subscale at Pre Survey. 


18 | Evaluation of the Student-Centered Arts-Learning Environments (SCALE) Project 


Results 


We first examined changes from Pre to Post Survey for each of the composite en- 
gagement subscales. Figure 5 illustrates that, in all cases, teachers’ perceptions of student 
engagement increased from Pre to Post Survey. The largest increase was evidenced in stu- 
dents’ behavioral engagement followed by cognitive engagement and affective engagement, 
respectively. Subsequent statistical analyses revealed the increases were not statistically sig- 
nificant (See Table 7). However, it is worth noting that the change in behavioral engagement 
approached statistical significance (p = .053). 


Figure 5. Student Engagement Subscale Scores (All Schools) 


5 


4.5 -—a03- 4.06 
4 


3.5 4 
3 
2.5 4 
2 
15 4 
1 


Affective Affective Behavioral Behavioral Cognitive Cognitive 
Pre Post Pre Post Pre Post 


— 
2 
3 
3 
= 
oO 
n 
os 
£ 
ov 
2 
3 
oO 
n 
ov 
1) 
i] 
£ 
Vv 
> 
<x 


Engagement dimension 


Table 7. Summary of Tests of Statistical Significance for Engagement Subscales (All Schools) 


Mean change (Post 


Survey average Standard Effect size 

minus Pre Survey error Sig (2- (Hedges’ 

Engagement dimension average) SD MEAN t df tailed) g*) 
Affective .02 .40 04 58 103 564 .07 
Behavioral .07 39 .04 1.96 103 .053 .20 
Cognitive .07 43 .04 1.55 102 .124 16 


*Ellis, P. D. (2009), "Effect size calculators," website: www.polyu.edu.hk/mm/effectsizefaqs/calculator/ 
calculator.html accessed on April 12, 2013. 


Next we examined each of the individual student engagement items on each subscale 
for significant changes over time. Two interesting findings emerged. First, teachers reported 
perceiving that students stayed on task more at Post Survey (M = 3.79, SD = .68) than at Pre 
Survey (M = 3.60, SD = .84). This difference was statistically significant t(103) = 2.756, p = 
.007. The effect size for this change was .25, a small positive effect. Second, teachers also re- 
ported significantly higher perceptions of student motivation at Post Survey (M = 3.90, SD = 
.66) than at Pre Survey (M = 3.75, SD = .80), t(100) = 2.137, p = .03. The effect size for this 
change was .20, a small positive effect. 


Post hoc analyses 
New and prior implementation schools 


Program staff informed us that some of the schools involved in the SCALE project 
had begun their initial implementation of arts integration activities during the 2012-2013 
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school year and that others had previously worked with the WVSO on this concept. We be- 
lieved this could be a potential source for variation among outcomes. As such, we conducted 
several post hoc analyses where we examined survey results for what we labeled new and 
prior implementation schools (i.e., those with either no history or at least some history of 
implementing arts integration activities with the WVSO, respectively). 


As with all schools, we first examined changes in the overall subscale scores for affec- 
tive, behavioral, and cognitive engagement. We found no statistically significant differences 
from Pre to Post Survey for new or prior implementation schools. However, new implemen- 
tation schools displayed positive changes in both behavioral and cognitive engagement that, 
though not statistically significant, exhibited effect sizes in excess of .25. This is a threshold 
considered to represent substantive importance or a “qualified positive effect” by the What 
Works Clearinghouse (What Works Clearinghouse, 2013). 


Second, we examined the individual items on each subscale. Three notable findings 
emerged. First, we found that teachers in prior implementation schools actually reported 
slightly lower perceptions of the extent to which students were excited about their school- 
work at Post Survey (M = 3.81, SD = .68) than at Pre Survey (M = 4.00, SD = .65). This dif- 
ference was statistically significant t(52) = -2.017, p = .049. The effect size for this change 
was -.28, a small negative effect. Second, we found that teachers in new implementation 
schools reported perceptions that students stayed on task more at Post Survey (M = 3.82, SD 
= .71) than at Pre Survey (M = 3.49, SD = .88). This difference was statistically significant 
t(50) = 2.832, p = .007. The effect size for this change was .41, a small positive effect. Third, 
teachers in new implementation schools reported that students were more likely to believe 
they were learning in their classes at Post Survey (M = 4.06, SD = .59) than at Pre Survey (M@ 
= 3.88, SD = .56). This difference was statistically significant, t(48) = 2.438, p = .01. The ef- 
fect size for this change was .31, a small positive effect. 


Higher and lower than average fidelity of implementation schools 


We also examined results for schools that had higher and lower than average imple- 
mentation fidelity scores based upon WVSO and WVDE personnel responses to the SCALE 
Project Implementation Rubric and Checklist. For this analysis, we divided the teacher sur- 
vey data into two groups. The first group included all survey responses from educators in 
those schools with a mean percent implemented greater than or equal to the median for all 
participating schools (85.9%). The second group included all survey responses from educa- 
tors in the remaining schools. We labeled these two groups, higher than average fidelity (HF) 
and lower than average fidelity (LF) schools, respectively. 


With respect to overall subscale scores, two interesting findings emerged. First, we 
found that educators in HF schools reported higher overall behavioral engagement among 
students at Post Survey (M = 3.74, SD = .36) than at Pre Survey (M = 3.62, SD = .40). This 
difference was statistically significant t(69) = 2.957, p = .004. The effect size for this differ- 
ence was .31, a small positive effect. Second, educators in HF schools also reported higher 
overall cognitive engagement among students at Post Survey (M = 3.66, SD = .50) than at 
Pre Survey (M = 3.54, SD = .52). This difference was statistically significant t(68) = 2.136, p 
= .03. The effect size for this difference was .23, a small positive effect. 
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For individual student engagement items, three interesting findings emerged. We 
found that educators in HF schools reported students stayed on task more at Post Survey (MW 
= 3.86, SD = .60) than at Pre Survey (M = 3.59, SD = .82). This difference was statistically 
significant t(69) = 3.453, p = .001. The effect size for this difference was .37, a small positive 
effect. Students also preferred more challenging assignments at Post Survey (M = 3.61, SD = 
.88) than at Pre Survey (M = 3.35, SD = .82). This difference was statistically significant 
t(68) = 2.495, p = .015. The effect size for this difference was .30, a small positive effect. 
Lastly, students were reported to be following instructions better at Post Survey (M = 3.96, 
SD = .53) than at Pre Survey (M = 3.81, SD = .60). This difference was statistically signifi- 
cant t(67) = 2.306, p = .024. The effect size for this difference was .26, a small positive effect. 


Table 8 summarizes the findings from our post hoc analyses for EQ 3. 


Table 8. Summary of Engagement Findings from Post Hoc Analyses 
Effect size/ 
Item Domain Finding Group direction 
Composite Score Behavioral Perceptions were higher Higher than Average Small positive 
Engagement at the end of the project Implementation Fidelity effect 
than at its outset. Schools 
Composite Score Cognitive Perceptions were higher Higher than Average Small positive 
Engagement at the end of the project Implementation Fidelity effect 
than at its outset. Schools 
Students are excited Affective Perceptions were lower Prior Implementation Small negative 
about their work at Engagement at the end of the project Schools effect 
school. than at its outset. 
Students stay on task. Behavioral Perceptions were higher New Implementation Small positive 
Engagement at the end of the project Schools effect 
than at its outset. Higher than Average 
Implementation Fidelity 
Schools 
Students follow Behavioral Perceptions were higher Higher than Average Small positive 
instructions in class. Engagement at the end of the project Implementation Fidelity effect 
than at its outset. Schools 
Students believe they Cognitive Perceptions were higher New Implementation Small positive 
are learning a lot in Engagement at the end of the project Schools effect 
their classes. than at its outset. 
Students prefer Cognitive Perceptions were higher Higher than Average Small positive 
challenging Engagement at the end of the project Implementation Fidelity effect 


assignments. 


than at its outset. 


Schools 
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Evaluation Question 4 


To what extent did the SCALE Project impact culture and climate in participating 


schools? 


We used four of the six subscales of the Perceptions of School Culture (POSC) ques- 
tionnaire (Cowley, Voelkel, Finch, & Meehan, 2006) as items embedded in the Project 
SCALE School Surveys (Pre Survey and Post Survey) to measure school culture and climate 
in SCALE schools. The subscales selected for this study included (a) collaborative working 


relationships (CWR, 13 items), (b) 
student responsibility for learning 
(SRFL, 12 items?), (c) teacher re- 
sponsibility for learning (TRFL, 13 
items), and (d) inviting physical en- 
vironment (IPE, 5 items). We first 
examined the reliability of the sub- 
scales at both Pre and Post Survey. 
We found that all were highly relia- 
ble (See Table 9). 


Table 9. Reliability Estimates for POSC Subscales 


Number of 
items on Pre survey Post survey 
POSC dimension subscale reliability* reliability* 
CWR 13 97 96 
SRFL 12 93 92 
TRFL 13 .96 96 
IPE 5 91 91 


*Cronbach’s Alpha (a) was used as the measure of reliability 


Next, we examined changes from Pre to Post Survey on all four POSC subscales. As is 
evidenced in Figure 6, the average scores for the SRFL and TRFL subscales increased slightly 
from Pre to Post Survey. However, the average scores for the CWR and IPE subscales de- 
creased slightly. Further analysis revealed that none of the changes was statistically signifi- 
cant (See Table 10). Additionally, none of the effect sizes observed for these changes met or 
exceeded the threshold for substantive importance (i.e., +.25). 


Figure 6. 
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2 This is normally a 13 item scale. When developing the survey, one item was unintentionally 
not included. There is a possibility that validity of this subscale measure may have been affected (see 
limitations section). However, there was so little change from Pre to Post Survey that it is unlikely that 
results or their interpretation were meaningfully affected. 
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Table 10. Summary of Tests of Statistical Significance for POSC Subscales (All Schools) 


Mean change (Post Standard Effect size 

survey average minus error Sig (Hedges’ 

Engagement dimension Pre survey average) SD MEAN t df (2-tailed) g*) 
CWR -.71 9.24 91 -.782 101 .432 .06 
SRFL 76 6.46 65 1.173 99 .243 .09 
TRFL 35 6.74 .68 510 97 .612 .04 
IPE -.03 8.28 .83 -.032 98 975 .002 


*Ellis, P.D. (2009), "Effect size calculators," website: www.polyu.edu.hk/mm/effectsizefaqs/calculator/ 
calculator.html accessed on April 12, 2013. 


Post hoc analyses 
New and prior implementation schools 


As with EQ3, we also conducted a series of post hoc analyses examining changes in 
POSC subscale means among respondents from new and prior implementation schools. Our 
analyses revealed no statistically significant changes in any of the subscale means from Pre 
to Post Survey for new or prior implementation schools. Additionally, none of the effect sizes 
for the changes observed met or exceeded the threshold for substantive importance (i.e., 
+.25). 


Higher and lower than average fidelity of implementation schools 


Our analyses revealed no statistically significant changes in any of the POSC subscale 
means from Pre to Post Survey for HF or LF schools. Additionally, none of the effect sizes for 
the changes observed met or exceeded the threshold for substantive importance (i.e., +.25). 


Evaluation Question 5 


To what extent did the SCALE Project result in improved lesson design and/or 

instructional delivery among participating schools? 

We constructed three measures of lesson design/instructional delivery for this study: 
(a) arts integration (i.e., integration of arts content into instruction, 6 items); (b) faculty/ 
community collaboration (i.e., collaboration among faculty role groups and community 
members in lesson design, 9 items); and (c) team-based collaboration (i.e., collaboration 
among school-based teams in lesson planning). As previously, each subscale was calculated 
by summing respondents’ responses to the individual items on the scale and then dividing by 
the total number of items. For each subscale, we first assessed the reliability at Pre and Post 
Survey. Table 11 indicates that all scales were reliable—exceeding the threshold of .70. 


Table 11. Reliability Estimates for Lesson Design/Instructional Delivery Subscales 


Lesson design/instructional delivery Number of items Pre Survey Post Survey 
dimension on subscale reliability* reliability 
Arts Integration 6 76 81 
Faculty/Community Collaboration 9 73 77 
Team-Based Collaboration 3 77 75 


*Cronbach’s Alpha (a) was used as the measure of reliability 
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As is evidenced in Figure 7, the average scores for all three lesson design/ 
instructional delivery subscales increased marginally from Pre to Post Survey. However, 
none of these changes was statistically significant (see Table 12). Additionally, none of the 
effect sizes for these changes exceeded the threshold for substantive importance (i.e., +.25). 


Figure 7. Lesson Design/Instructional Delivery Subscale 
Scores at Pre- and Post-Test 
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Table 12. Summary of Tests of Statistical Significance for Lesson Design/Instructional Delivery Subscales 


(All Schools) 
Mean change (Post 

Survey average Standard Effect size 
minus Pre Survey error Sig (2- (Hedges’ 
Engagement dimension average) SD MEAN t df tailed) g*) 
Arts Integration .07 59 .06 1.153 103 .25 .08 
Faculty/Community 10 79 .08 1.304 103 19 15 

Collaboration 
Team-Based .01 .65 .06 .201 103 .84 .01 


Collaboration 


*Ellis, P.D. (2009), "Effect size calculators," website: www.polyu.edu.hk/mm/effectsizefaqs/calculator/ 
calculator.html accessed on April 12, 2013. 


Next, we examined each of the individual items on the three lesson design/ 
instructional delivery subscales. One interesting finding emerged. Teachers reported more 
integration of dance/movement strategies into instruction at Post Survey (M = 3.51, SD = 
1.04) than at Pre Survey (M = 3.27, SD = 1.10). This difference was statistically significant 
t(96) = 2.826, p = .006. The effect size for this change was .22, a small positive effect. 


Post hoc analyses 
New and prior implementation schools 


As with previous analyses, we examined the lesson design/instructional delivery sub- 
scale means and individual subscale items for both prior and new implementation schools. 
While no significant findings emerged for overall subscale means, two interesting findings 
emerged with respect to individual subscale items. First, new implementation schools re- 
ported more collaboration with community members in the development of their lessons at 
Post Survey (M = 2.49, SD = .87) than at Pre Survey (M = 2.18, SD = .75). This difference 
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was statistically significant t(48) = 2.335, p = .02. The effect size for this change was .38, a 
small positive effect. Second, new implementation schools also reported more integration of 
dance/movement strategies into their own instruction at Post Survey (M = 3.54, SD = 1.07) 
than at Pre Survey (M = 3.15, SD = 1.23). This difference was statistically significant t(45) = 
3.564, p = .001. The effect size for this change was .19, a small positive effect. 


Primary content area taught 


We posited that there may be differences among general educators and arts educa- 
tors in the extent to which they reported integration of arts content and collaboration with 
other faculty/community members. Therefore, for EQ5, we conducted a series of additional 
post hoc analyses disaggregating the data by respondents’ primary content area. We ulti- 
mately examined the results for two groups of educators: arts and elementary education 
teachers. 


First, we examined changes in the overall lesson design/instructional delivery sub- 
scale means. We found no statistically significant differences from Pre to Post Survey for ei- 
ther group of educators. However, when examining individual subscale items two interesting 
findings emerged. First, elementary educators reported more integration of dance/ 
movement strategies into their own instruction at Post Survey (M = 3.56, SD = 1.04) than at 
Pre Survey (M = 3.32, SD = 1.10). This difference was statistically significant t(62) = 2.258, p 
= .02. The effect size for this change was .22, a small positive effect. Second, elementary edu- 
cators also reported more collaboration with arts teachers at Post Survey (M = 2.31, SD = 
1.07) than at Pre Survey (M = 2.06, SD = 1.05). This difference was statistically significant 
t(63) = 2.646, p = .01. The effect size for this change was .23, a small positive effect. 


Higher and lower than average fidelity of implementation schools 


First, we examined changes in the overall lesson design/instructional delivery sub- 
scale means. We found no statistically significant differences from Pre to Post Survey for HF 
or LF schools. However, when examining individual subscale items, two interesting findings 
emerged. Educators in HF schools reported less integration of creative writing instructional 
strategies at Post Survey (M = 3.46, SD = 1.11) than at Pre Survey (M = 3.69, SD = .93). This 
difference was statistically significant t(67) = -2.241, p = .03. The effect size for this differ- 
ence was -.22, a small negative effect. Educators in HF schools also reported more integra- 
tion of dance instructional strategies at Post Survey (M = 3.45, SD = 1.06) than at Pre Survey 
(M = 3.16, SD = 1.13). This difference was statistically significant t(68) = 2.921, p = .005. The 
effect size for this difference was .26, a small positive effect. 


A summary of the findings from post hoc analyses related to lesson design/ 
instructional delivery can be found in Table 13. 
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Table 13. Summary of Lesson Design/Instructional Delivery Findings from Post Hoc Analyses 


Effect size/ 


Item Domain Finding Group(s) direction 

In the most recent school = Faculty/Community Perceptions were New Small positive 
year, how often did you Collaboration higher at the end of Implementation effect 
collaborate with the the project than at Schools 

following people in its outset. 


designing lessons for your 
own class? (Community 


Members) 
Inthe most recent school = Arts Integration Perceptions were New Small positive 
year, how often did you higher at the end of Implementation effect 
integrate any of the the project than at Schools 
following strategies in your its outset. Elementary 
classroom lessons? (Dance/ Educators 
Movement) Higher than 
Average Fidelity of 
Implementation 
Schools 
Inthe most recent school Arts Integration Perceptions were — Higher than Small negative 
year, how often did you lower at the end of Average Fidelity of effect 
integrate any of the the project than at Implementation 
following strategies in your its outset. Schools 
classroom lessons? (Creative 
Writing) 
In the most recent school = Faculty/Community Perceptions were Elementary Small positive 
year how often did you Collaboration higher at the end of Educators effect 
collaborate with the the project than at 
following people in its outset. 


designing lessons for your 
own class? (Art Teacher) 
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Evaluation Question 1 


What was the quality of the training and technical assistance that was provided to 

SCALE Project schools? 

Professional development offered to participants in the SCALE Project was well at- 
tended and well received. The overall ratings were remarkably high, with participants 
strongly agreeing about the high quality of the training and trainers on multiple measures, 
and approaching strong agreement about the high quality of the materials provided. The 
comments from participants also strongly praised the program, materials, and trainers. A 
few indicated they did not feel prepared to implement the project in their schools—that is, 
they still had lots of questions—after the summer professional development, but when they 
had completed all three training events, they indicated they were looking forward to using 
what they had learned back in their schools and classrooms. Overall, based on participant 
perceptions, SCALE Project team leaders were well trained to facilitate the program in their 
schools. 


Notably, the training feedback survey provided some compelling evidence that the 
alignment of the SCALE professional development opportunity to school-wide goals was 
lacking—approximately 40% of respondents held this view. While this may seem alarming at 
first, we must consider the fact that, as one participant commented, school-wide goals may 
not be designed to be holistic or considerate of all factors impacting school quality. This as- 
sertion is particularly interesting when one considers that most schools implementing the 
SCALE program are historically low achieving and serve large percentages of disadvantaged 
students. In these schools, it is probable that the school-wide goals are focused heavily upon 
improving test performance in order to move the school out of improvement status. One 
consequence of this decision may be that the schools have altogether or partially eschewed 
potentially important interventions such as arts integration. If this assumption is true, it is 
especially salient given the generally positive results we observed in this study with respect 
to student engagement. Specifically, we found that some teachers perceived higher student 
cognitive and behavioral engagement at the conclusion of this project than at its outset. 
These outcomes are arguably critical intermediate outcomes that must occur before whole 
school achievement can improve. Thus, arts integration or other holistic intervention strate- 
gies could be an integral part of comprehensive school improvement efforts. 


Evaluation Question 2 
How well did the SCALE Project schools implement the central components of the 
program? 


Rubric and benchmark data provided by West Virginia Symphony Orchestra (WVSO) 
and West Virginia Department of Education (WVDE) personnel indicated that in most 
schools, the SCALE Project was well implemented. As a group, rubric data indicated that 
schools saw the greatest level of implementation relative to student engagement in the arts 
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and the lowest level of implementation relative to improving lesson design. Other aggregated 
benchmark data showed the following: 


e There was agreement or strong agreement that student engagement was high on all 
measures. 


e In the large majority of schools, at least six of 10 role groups were involved in the 
project, with Title I teachers, physical education teachers, parents, and community 
members the most commonly left out. 


e The arts were well integrated into the SCALE school projects, with two schools also 
integrating technology. 


e The large majority of schools implemented eight of nine major SCALE Project com- 
ponents, with one component—regular SCALE team meetings—the component least 
commonly observed. 


Across all rubric and benchmark measures, the SCALE project saw about 80% implementa- 
tion in the 15 schools for which we obtained rubric data.3 


Implementation was far from even, however. Four schools had implementation rates 
above 90% (high-implementation schools), while four had implementation rates below 75%, 
ranging down to 51% (low-implementation schools). For those that had lower levels of im- 
plementation, several issues seemed to be the most common. There was difficulty forming a 
team and arranging for it to meet regularly. In three of the four low-implementation schools, 
there was limited arts integration professional development for staff. In three, there was lim- 
ited involvement of content areas other than the arts, while in two the arts teachers were not 
involved. Half of the low-implementation schools involved only some, not all, of their class- 
rooms. 


The unevenness in implementation did not seem related to the schools’ tenure in the 
program. All four low-implementation schools had previous experience with the SCALE Pro- 
ject, as did three of the four high-implementation schools. Further, four of the seven schools 
with higher than the median implementation scores were new to the project this year (2012- 
2013). Neither did the level of implementation appear related to the schools’ involvement in 
the School Improvement Grant (SIG) program, as half of the low implementation and half of 
the high implementation schools were SIG schools. 


Although it is not possible to say with certainty, successful implementation seemed 
more related to the school’s commitment to the project—which included establishing an ac- 
tive, multidisciplinary team, strong collaboration, and involvement of the whole school— 
rather than to other possible factors (e.g., lack of experience or SIG status). Some schools 
may have volunteered for the project simply to give their students the opportunity to go toa 
WVSO concert (a definite benefit), with less enthusiasm for learning new approaches to les- 


3 WVDE and WVSO staff were not able to arrange a final observation visit at one school, 
Smoot Elementary, despite multiple attempts. 
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son design, cross disciplinary collaboration, and arts integration across the curriculum. Oth- 
er schools enthusiastically engaged in the whole vision for the project. 


Evaluation Question 3 


To what extent did the SCALE Project result in increased student engagement in 

music and the arts and in other content areas? 

SCALE school teacher surveys conducted before and after implementation of the pro- 
ject contained three subscales measuring educators’ perceptions of student affective, behav- 
ioral, and cognitive engagement in the arts and other content areas. All three subscales 
showed small improvements, although only one, behavioral engagement, approached statis- 
tical significance. When looking at individual survey items that made up the subscales, two 
interesting findings emerged—teachers reported small positive effects for students staying 
on task and for student motivation. 


When comparing schools with prior experience in SCALE with those new to the pro- 
ject, there were small but substantively important positive changes in both behavioral and 
cognitive engagement for schools new to SCALE. Looking at individual items, we found two 
of the strongest effects we detected in the study: Teachers in new-implementation schools 
reported higher levels of students staying on task at Post Survey than at Pre Survey and 
higher levels of students believing they were learning in their classes. For prior- 
implementation schools we found there were slightly lower perceptions of the extent to 
which students were excited about their schoolwork at Post Survey. This is interesting be- 
cause in several of the schools the WVSO and WVDE technical assistance providers noted 
that after the project ended teachers slipped back into more familiar modes of instruction, 
and away from the more collaborative and integrative approach they strived for during the 
project. 


As noted in the discussion of EQ2, there were large differences among the schools in 
their levels of implementation, so we looked at what bearing, if any, implementation fidelity 
had on changes between Pre and Post Survey. With respect to overall subscale scores, we 
found that educators in high-fidelity schools reported both higher overall behavioral and 
cognitive engagement among students at Post Survey than at Pre Survey. When we looked a 
little deeper at individual student engagement items, educators in high-fidelity schools at 
Post Survey reported students (a) stayed on task more, (b) preferred more challenging as- 
signments, and (c) followed instructions better. 


Evaluation Question 4 


To what extent did the SCALE Project impact culture and climate in participating 

schools? 

Our analysis of pre- and posttest responses to school culture items on the school sur- 
vey revealed no statistically significant or substantively important changes from pre- to post- 
test data collections on any of the four subscales of the Perceptions of School Culture survey 
items (Cowley, Voelkel, Finch, & Meehan, 2006), including (a) collaborative working rela- 
tionships, (b) student responsibility for learning, (c) teacher responsibility for learning, and 
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(d) inviting physical environment. These findings held when we disaggregated the responses 
by new or prior implementation and by high- and low-implementation fidelity. 


This is not surprising for such a brief intervention such as the SCALE Project. It is 
possible that results for these subscales could change with involvement over multiple years, 
but for one intervention that lasted a few months, it would be unusual to see significant 
changes. Furthermore, our analysis was limited to examining the entire group of participat- 
ing schools in aggregate. Due to low sample sizes within each school, we were unable to as- 
sess if there were statistically significant improvements in individual school’s culture. Yet we 
do have anecdotal evidence from the implementation rubric and checklist that some schools 
saw improvements. A principal “mentioned the level of camaraderie in the teachers’ lounge 
was "buzzing" with flow of ideas and support for fellow teachers,” which “translated directly 
into student enthusiasm for learning throughout the course of the project.” Additionally, a 
school transformation specialist commented specifically on “positive culture change he had 
witnessed in this school,” which he partly attributed to the SCALE Project, giving “the im- 
pact high marks.” 


Evaluation Question 5 


To what extent did the SCALE Project result in improved lesson design and/or 

instructional delivery among participating schools? 

We constructed three subscales in the Project SCALE School Survey (Pre and Post 
Survey versions) for lesson design and instructional delivery to measure (a) integration of 
arts content into instruction, (b) collaboration among faculty role groups and community 
members in lesson design, and (c) collaboration among school-based teams in lesson plan- 
ning. We found no statistically significant or substantively important changes at the subscale 
level for the whole group, for schools disaggregated by experience with the SCALE Project 
(new versus prior), nor for schools disaggregated by level of fidelity of implementation (high 
versus low). We also disaggregated responses by content area (arts versus other content are- 
as), and again did not find changes to report. When looking at individual items, however, 
there were several interesting—albeit small effects at Post Survey: 


e Teachers from new-implementation schools reported 


o more collaboration with community members in the development of their les- 
sons, and 


o more integration of dance/movement strategies into their own instruction. 
e Non-arts elementary teachers reported 
o more integration of dance/movement strategies into their own instruction, and 
o more collaboration with arts teachers. 
e Teachers from high-implementation-fidelity schools reported 
0 less integration of creative writing instructional strategies, and 
o more integration of dance instructional strategies. 
The findings about dance/movement strategies are interesting, but may reflect a change 
from little or no involvement in these strategies to occasional use. This finding is also inter- 
esting given that WVSO/WVDE representatives rated dance/movement strategies as the 
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least integrated of the creative strategies. The reduction in creative writing strategies may 
reflect an expansion to other forms such that creative writing is sharing the stage with other 
instructional strategies introduced through the SCALE project. 


Limitations of the Study 


Several limitations of this study bear mentioning. First, all data are self-reported and 
thus subject to various threats to validity. One such threat that could have been introduced is 
termed social desirability bias. This occurs when respondents provide overly positive re- 
sponses to a survey or questionnaire due to their desire to be viewed favorably. In this case, 
we asked educators about their own instructional practices and their contribution to a 
school-wide project. This could have been an incentive to report more positive results. Addi- 
tionally, our survey was voluntary in nature and thus nonresponse bias could also be a fac- 
tor. This occurs when respondents who elect not to participate in a survey differ in a 
meaningful way with those who do with respect to the outcome being measured. For exam- 
ple, it is possible that individuals with more negative feelings about the project elected not to 
participate in the survey. This type of bias can inflate results. To some extent, these are po- 
tential issues in all studies that employ self-report measures. In our study we have no way of 
knowing the extent to which these issues impacted results—especially on the Pre and Post 
Surveys, which had lower response rates than the professional development survey. 


We encountered one technical difficulty when we unintentionally excluded an item in 
one of the subscales drawn from the POSC instrument. To compensate for the missing item, 
we had to apply a modified multiplier to the subscale. As a result, there is a small possibility 
that the validity of the SRFL measure was affected. However, as we observed little or no 
change in this and other POSC subscales, we do not anticipate this error impacted our find- 
ings. 


Another limitation of our study involves our inability to draw school-level conclu- 
sions regarding the impact of the various projects implemented by schools. Because we did 
not receive a large number of completed surveys from many schools, we were unable to con- 
fidently disaggregate results by school. Instead, we chose to aggregate our results to examine 
specific groups of schools or educators (e.g., all schools, schools by implementation status, 
arts educators, etc.). At best, this approach only allows us to ascertain program-level results. 
It does not afford an opportunity to examine outcomes for each individual school. This is an 
issue because qualitative data related to implementation fidelity indicated there may have 
been a good deal of variation across schools in terms of their projects and the contexts in 
which they were implemented. It is reasonable to assume that some of the study outcomes 
could have differed by school. Unfortunately we were unable to test this hypothesis, and 
what we are left with is an evaluation of the SCALE program in which exceptional cases may 
have gone undocumented. 
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Recommendations 


We make the following recommendations based upon our results: 


e To the extent possible, we recommend continuing this project. Educators appear to 
perceive positive benefits of the program for their students especially in the areas of 
cognitive and behavioral engagement. These are important outcomes that could lead 
to improved student achievement if sustained. 


e In this study, we found that new implementation schools realized more positive out- 
comes than prior-implementation schools. The excitement factor in new schools 
could have contributed to these findings. Efforts should be made to sustain initial ex- 
citement so that prior implementation schools can continue to realize benefits. 


e We also found that higher-than-average implementation fidelity schools experienced 
more positive outcomes than those schools that did not implement many compo- 
nents with fidelity. We found no significant changes for lower-than-average imple- 
mentation schools. That is, while failure to implement the program as intended is not 
necessarily associated with negative outcomes, it does potentially maintain the status 
quo. Program staff should use these results as a catalyst for participating schools, to 
illustrate that a school’s level of commitment can make or break the project. 


e Ensure that schools participating in the project build in sufficient common planning 
time to accomplish the necessary collaboration. This time is essential to ensure the 
school’s project is implemented with fidelity and achieves the intended school-wide 
outcomes. Administrator support is critical in this regard and should be discussed 
early on in the project. 


e Develop strategies to ensure that once the school project concludes, the faculty does 
not return to business as usual. One strategy may be finding ways to sustain the mo- 
mentum of the project—that is to continue on with other collaborative projects that 
integrate various content areas, including the arts. Another strategy may involve ad- 
dressing beliefs that time spent on a project such as SCALE is time taken away from 
improving test scores in mathematics and reading/language arts. Helping educators 
understand the strong connections between high student cognitive and behavior en- 
gagement—as seen in most of the SCALE schools this year—and high student 
achievement could help reduce the anxiety felt about making the sorts of changes in 
lesson planning and instruction that the SCALE project encouraged. 
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Guidance for SCALE Team Leaders/Mentors on the Collection 
of Faculty E-Mail Addresses 


e As part of the SCALE Project, we are conducting an evaluation. We will be disseminating 
two surveys to your school via e-mail. As such, we are requesting that you provide us 
with e-mail addresses where we can reach your faculty. 


e Please collect the information indicated in Table 2 below for each individual in your 
school who regularly interacts with students in a learning environment (e.g., teachers). 
You do not need to include administrators on this list. You will provide the 
list to the WVDE in by Friday September 14, 2012. 

1. Avalid e-mail address that the individual checks regularly 


2. An indicator of whether or not each person serves or will serve on the school’s 
SCALE project team 


e Your list of email addresses does not need to include anything but the school name, a list 
of e-mail addresses, and an indication of whether or not the individual serves on the 
SCALE team at your school. Please see an example, below. 


School Name: Wild and Wonderful High School Serves on SCALE Team? 


Teacher1@access.k12.WV.Us 


Teacher2@gmail.com YES 
| Teacher3@yahoo.com | YES | 


e Please note: If you already have a list of e-mail addresses for the faculty in 
your central office, you may provide that list to us rather than creating a new 


one, but we ask that you do the following: 


1. Cross out the names/email-addresses for staff who do not regularly interact 
with students in a learning environment (e.g., service personnel) and 
administrators 


2. Place an asterisk (*) or a star next to the e-mail addresses of any staff 
members who serve on the school’s SCALE project team 


e You will send your completed list of e-mail addresses to Jack Deskins by Friday 
September 14, 2012. (jdeskins @access.ki2.wv.us). 
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Appendix C. Survey Instruments 


SCALE Project Professional Development Evaluation Survey 


Project SCALE Professional Development Evaluation Survey [WWV/DE- 


Welcome to the Project SCALE Professional Development Evaluation Survey 


Welcome and thank you for agreeing to take part in this important data collection activity. 


By participating in this data collection activity, you are consenting to participate in a research study, the purpose of 
which is to determine the effectiveness of an educational program. The results of this research study will also be used to 
help determine revisions to the program and possibly to inform the development of future education programs. Your 
participation in this study shall be limited to completing a brief survey and should not take more than 20 minutes of your 
time. You will be presented with a series of items and asked to indicate your responses by either providing a rating or a 
short open-ended response. 


Participation in this research study poses no more risk than that which you would encounter during the course of a 
normal day. While you may not experience tangible direct benefits from participating in this research study, indirect 
benefits may include the program being better tailored to meet your needs and/or improved outcomes for study 
participants and/or students. All responses to this data collection activity shall remain completely confidential and no 
identification of individual participants will occur. All results shall be reported in aggregate. 


There is no compensation for taking part in this research study. Please note that your participation is completely 
voluntary and may be discontinued at any time. No penalties or loss of benefits to which you are otherwise entitled shall 
occur should you refuse to participate in the research. If you have questions about Project SCALE you may contact Jack 
Deskins, Coordinator at the WDE Office of Instruction, at 304.558.5325 or jdeskins@access.k12.wv.us. Additionally, if 
you have questions about the research associated with the program or this data collection activity, you may contact Pat 
Hammer, Coordinator, at the WVDE Office of Research at 304.558.2546 or phammer@access.k12.w.us. 


This research study has been reviewed and approved by the West Virginia Department of Education (WDE) Institutional 
Review Board (IRB). Should you desire additional details about this study's approval status, you may contact the WDE 
IRB chairperson, Nathaniel Hixson (nhixson@access.k12.wv.us). 


Again, thank you for your participation! 
I. Please Tell us About Yourself. 


Please tell us the county where you work. 


Please select one. [tid 


Please indicate your school. 


What grade level do you teach? (Select all that apply) 
(ile 
[_]s 
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Project SCALE Professional Development Evaluation Survey [WVDE- 


How many years of experience have you had in your current position? 


Is your primary content area: 
O Arts (Music, Dance, Theater, Visual Art) 
O Physical Education 

O Elementary Education 

O Special Education 


O Title | 


Other (please specify) 


Which of the following did you attend? 


[] June 2012 SCALE Training (INSERT LOCATION OF TRAINING HERE) 


[| August 2012 SCALE Training Follow-up Webinar 


Il. Your Evaluation of the PD. 
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Project SCALE Professional Development Evaluation Survey [WWVDE- 


Please indicate the level to which you agree or disagree with the following statements 
about the training you received as part of Project SCALE. 


Strongly . Strongly 
Neutral Disagree 7 
Disagree 
The training was high quality. 
The training was relevant. 
The training was well organized. 
The training was specific and content-focused. 


The training was hands-on and included active learning opportunities. 


Training objectives were clearly stated before sessions began. 


OOOOOO0O?; 


Training sessions began and ended in a timely fashion. 


Which of the following statements best describes the usefulness of the PD you received 
as part of Project SCALE? 


O It was a good start. 

O It was a good start, but | have a lot of questions. 

O It was a good start, and | look forward to using what | learned in my classroom (or work setting). 
O It provided everything | need to use what | learned in my classroom (or work setting). 


O | don't think that these ideas will work very well in my classroom (or work setting). 


& It's too early to tell. 


Indicate the extent to which the PD you received as part of Project SCALE met your 
professional needs. 


CD It addressed my professional learning needs completely. 
O It addressed some of my professional learning needs. 
O It did not address my professional learning needs. 


O This professional development did not help much because | already know what | need to know about this topic. 


Which of the following statements best describes the likelihood that you will apply what 
you learned in this PD in your classroom (or work setting)? 


C) | already practice/apply the knowledge/skills this training provided in my classroom (or work setting), and it seems to work well. 


O | have already practiced/applied the knowledge/skills this training provided in my classroom (or work setting), but it is not appropriate for 


my students. 


O | look forward to practicing/applying the knowledge/skills in my classroom (or work setting) during the upcoming school year. 


O | don’t think what | learned here will work for my students so | don’t envision applying the knowledge/skills. 
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To what extent was the PD/training aligned with your school’s/program’s goals for 
improving instruction? 


O The PD was VERY CLOSELY aligned with school’s/program’s goals for instructional improvement. 
O The PD was SOMEWHAT aligned with school’s/program’s goals for instructional improvement. 
O The PD was NOT ALIGNED with school’s/program’s goals for instructional improvement. 


O The PD was INCONSISTENT with school’s/program’s goals for instructional improvement. 


O | don’t know. 


Which of the following statements best describes how the Project SCALE PD opportunity 


you attended compares with other PD opportunities in which you have participated in the 
last three years? 


O This professional development was MORE USEFUL than other professional development that | have participated in. 


O This professional development was ABOUT THE SAME AS other professional development that | have participated in. 


O This professional development was LESS USEFUL than other professional development that | have participated in. 


O | don't have an opinion. 


O | haven't participated in any other professional development in the last three years. 


Please provide any additional comments/feedback you may have about this section. 


a 


Ill. Your Evaluation of PD Materials. 


Please indicate the level to which you agree or disagree with the following statements 
about PD materials/resources. 


7 Strongly 
Neutral Disagree 


Disagree 
Adequate amounts of training materials/resources were provided. 


Materials/resources were relevant to my work. 


The materials/resources provided were of high quality (i.e., based on 
recent research and evidence-based). 


The materials/resources provided were useful to my work. 


Please provide any additional comments/feedback you may have about PD 
materials/resources. 
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Project SCALE Professional Development Evaluation Survey [WV/DE- 


Please indicate the level to which you agree or disagree with the following statements 
about the trainer(s) who delivered the PD. 


2 Strongly 

Neutral Disagree . 
Disagree 

Trainer(s) were knowledgeable about the topic. 

Trainers(s) were well organized. 

Trainers(s) presented the material clearly and effectively. 

Trainers(s) facilitated discussions well. 


Trainers(s) answered questions raised during sessions adequately. 


Please provide any additional comments/feedback you may have about the trainer(s). 


4 


IV. Any Additional Feedback? 


Please provide us with any additional feedback you might have. 


ee 


What (if any) are your remaining PD/technical assistance needs with respect to Project 
SCALE? 


Thank you very much for participating in this survey. Please click "done" to submit your response. 


V. You are done! 
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Project SCALE Implementation Rubric and Checklist [WWVDE-CIS-99] 


Welcome to the Project SCALE Implementation Rubric and Checklist 


Welcome and thank you for agreeing to take part in this important data collection activity. 


By participating in this data collection activity, you are consenting to participate in a research study, the purpose of 
which is to determine the effectiveness of an educational program. The results of this research study will also be used to 
help determine revisions to the program and possibly to inform the development of future education programs. Your 
participation in this study shall be limited to completing a brief survey and should not take more than 20 minutes of your 
time. You will be presented with a series of items and asked to indicate your responses by either providing a rating or a 
short open-ended response. 


Participation in this research study poses no more risk than that which you would encounter during the course of a 
normal day. While you may not experience tangible direct benefits from participating in this research study, indirect 
benefits may include the program being better tailored to meet your needs and/or improved outcomes for study 
participants and/or students. All responses to this data collection activity shall remain completely confidential and no 
identification of individual participants will occur. All results shall be reported in aggregate. 


There is no compensation for taking part in this research study. Please note that your participation is completely 
voluntary and may be discontinued at any time. No penalties or loss of benefits to which you are otherwise entitled shall 
occur should you refuse to participate in the research. If you have questions about Project SCALE you may contact Jack 
Deskins, Coordinator, at the VWWDE Office of Instruction at 304.558.5325 or jdeskins@access.k12.wv.us. Additionally, if 
you have questions about the research associated with the program or this data collection activity, you may contact Pat 
Hammer, Coordinator, at the WVDE Office of Research at 304.558.2546 or phammer@access.k12.w.us. 


This research study has been reviewed and approved by the West Virginia Department of Education (VWWDE) Institutional 
Review Board (IRB). Should you desire additional details about this study's approval status, you may contact the WWDE 
IRB chairperson, Nathaniel Hixson (nhixson@access.k12.wv.us). 


Thank you for your participation in this important effort! 
Section I: Please tell us about the school about which you are reporting. 


1. Please select the school from the list below. 


[| 


Section Il: Description of the School's SCALE Project 
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Project SCALE Implementation Rubric and Checklist [WVDE-CIS-99] 
2. Please briefly describe this school's SCALE project (What was/were the project's main 
goal(s)? Who was involved in planning and implementing the project? How did it involve 
the arts? What content areas were included?, etc.). 


Section II: Description of the School's SCALE Project 


3. Which of the following groups of educators were involved in this school’s SCALE 
project? 


Music teacher 

Art teacher 

Physical education teacher 
Title | teacher 

General classroom teachers 
Special education teachers 
Administrators 

Community members 


Parents 


O O 
O O 
O O 
O O 
O O 
O O 
O O 
O O 
O O 
O O 


WvSO 


Other (please specify) 


————————aaeaaaa 


Section II: Description of the School's SCALE Project 
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Project SCALE Implementation Rubric and Checklist [WVDE-CIS-99] 
4. Which of the following were integrated into the school's SCALE project? 


Music 

Drama 

Visual art 
Creative writing 
Dance/movement 


Other (please specify) 


Section Il: Description of the School's SCALE Project 


5. Through the project, students were able to: 


Strongly Disagree Disagree 


> 
a 

cS 

o 


Strongly Agree 


Connect content between/among multiple disciplines 
Represent complex ideas 

Share created products with others 

Participate fully as individuals 

Collaborate meaningfully with each other 

Participate in higher-order learning conversations 


Engage meaningfully in the arts 


OQOOOQO00O0O 
OQOQOOO0O00O 
OQOOOO000O 


Work collaboratively with the WVSO 


Section III: Core Components of SCALE (Student Engagement in Learning) 


6. Please indicate the extent to which this school’s SCALE project was engaging to 
students, by choosing the most accurate description. 


O As a result of the project, students exhibit a higher level of engagement in the classroom, not only physically, but through demonstration 


of cognitive engagement with the content and affective engagement in the learning process, driven by student inquiry. 


O As aresult of the project, students exhibit a higher level of physical engagement in the classroom. There is some evidence of cognitive 


and/or affective engagement, driven by student inquiry. 


O As aresult of the project, students exhibit a higher level of physical engagement in the classroom. There is minimal evidence of 


cognitive and/or affective engagement, and limited indication of student inquiry. 


O There is little evidence of physical, cognitive, or affective engagement in the classroom. 


7. Explain the evidence upon which you based your rating above. 


tas 
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Project SCALE Implementation Rubric and Checklist [WVDE-CIS-99] 
Section Ill: Core Components of SCALE (Student Engagement in the Arts) 


8. Please indicate the extent to which the school's SCALE project engaged students in the 
arts, by selecting the most accurate description. 


O As aresult of the project, students exhibit a higher level of engagement in the classroom, not only physically, but through demonstration 
of cognitive engagement with the content and affective engagement in the learning process, driven by student inquiry. 


O As aresult of the project, students exhibit a higher level of physical engagement in the classroom. There is some evidence of cognitive 
and/or affective engagement, driven by student inquiry. 


O As aresult of the project, students exhibit a higher level of physical engagement in the classroom. There is minimal evidence of 
cognitive and/or affective engagement, and limited indication of student inquiry. 


O There is little evidence of physical, cognitive, or affective engagement in the classroom. 


9. Explain the evidence upon which you based your rating above. 


~~ 


Section III: Core Components of Project SCALE (Collaboration for Arts Integ... 


10. Please indicate the extent to which the school's SCALE project exhibited collaboration 


for arts integration and other interdisciplinary learning, by choosing the most accurate 
description. 


O As a result of the project, classroom teachers use standards-based arts integration as a regular teaching strategy and, when applicable, 
collaborate with school arts teachers to design instruction. 


O As a result of the project, classroom teachers use standards-based arts integration as a sporadic teaching strategy and occasionally 
collaborate with school arts teachers to design instruction. 


O There is evidence of attempted arts-integration and collaboration with school arts teachers, but little attention to standards-based 
instruction. 


O There is little evidence of either standards-based arts integration as a teaching strategy, or collaboration with school arts teachers. 


11. Explain the evidence upon which you based your rating above. 


a 


Section Ill: Core Components of Project SCALE (Lesson Design and Interdisci... 
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Project SCALE Implementation Rubric and Checklist [WWVDE-CIS-99] 


12. Please indicate the extent to which the school's SCALE project exhibited 
improvements in lesson design, by choosing the most accurate description. 


O As a result of the project, teachers personalize learning to a greater degree; use student inquiry as a primary instructional method; 
promote greater student collaboration; and demonstrate a deep understanding of standards-based instructional design, including learning 
experiences aligned to standards, teaching to mastery, designing for engagement, and acceptable evidence of learning. 


O As a result of the project, teachers have made efforts to personalize learning; use student inquiry as an occasional instructional method; 
promote periodic student collaboration; and demonstrate an increased understanding of standards-based instructional design, including 
learning experiences aligned to standards, teaching to mastery, designing for engagement, and acceptable evidence of learning. 


O As aresult of the project, teachers have made some effort to personalize learning, though there is little evidence of student inquiry and 


collaboration as instructional methods. Teachers demonstrate some fundamental misunderstandings of standards-based instructional design. 


C) There is little evidence of improvement in instructional design as a result of this project. 


13. Explain the evidence upon which you based your rating above. 


a 


Section Ill: Core Components of Project SCALE (School Culture) 
14, Please indicate the extent to which the school's SCALE project has resulted in 


improvements in school culture. 


= As a result of the project, a greater number of faculty members are working together; both students and teachers have taken responsibility 


for learning; the physical environment of the school has become more inviting; and students are allowed greater latitude in decision making. 


O As a result of the project, some faculty members are working together; both students and teachers have taken an increased responsibility 


for learning; the physical environment of the school has become somewhat more inviting; and students are occasionally allowed to make 
decisions regarding their own learning. 


O As a result of the project, a few faculty members are working together; students and teachers in some classrooms have taken an increased 


responsibility for learning; there are small improvements to the physical environment of the school; and some students are allowed to make 
small decisions regarding their own learning. 


O There is little evidence of improvement in school culture as a result of this project. 


15. Explain the evidence upon which you based your rating above. 


Section IV: Implementation Checklist 
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Project SCALE Implementation Rubric and Checklist [WVDE-CIS-99] 
16. Which of the following are true of this school's SCALE project? 


Yes 
A Project SCALE team leader was identified in this school 
A Project SCALE team was identified in this school 


The Project SCALE team consisted of teachers from a variety of content 
areas 


The Project SCALE team included teachers of the arts 


The Project SCALE team met regularly to discuss the school's SCALE 
project 


Professional development related to arts integration was provided for all 
staff 


All classes participated in the school's SCALE project 


Information about the school's SCALE project was communicated in a 
timely fashion to all staff 


Support materials were shared with all staff 


O OO O OO OOO: 


17. Include any other information that would provide context for the level of 
implementation that took place at this school (optional). 


a 


Section V: Any Additional Feedback 


18. Please provide any additional comments you have about Project SCALE at this school. 


a 


The Survey is Complete! 


Thank you for your time. Please click "done" to submit your response. 
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Project SCALE School Survey (Pre Survey) 


Instrument 


Project SCALE School Survey [VWWDE-CIS-94] 
Welcome to the Project SCALE School Survey 


Welcome and thank you for agreeing to take part in this important data collection activity. 


By participating in this data collection activity, you are consenting to participate in a research study, the purpose of 
which is to determine the effectiveness of an educational program. The results of this research study will also be used to 
help determine revisions to the program and possibly to inform the development of future education programs. Your 
participation in this study shall be limited to completing a brief survey and should not take more than 20 minutes of your 
time. You will be presented with a series of items and asked to indicate your responses by either providing a rating or a 
short open-ended response. 


Participation in this research study poses no more risk than that which you would encounter during the course of a 
normal day. While you may not experience tangible direct benefits from participating in this research study, indirect 
benefits may include the program being better tailored to meet your needs and/or improved outcomes for study 
participants and/or students. All responses to this data collection activity shall remain completely confidential and no 
identification of individual participants will occur. All results shall be reported in aggregate. 


There is no compensation for taking part in this research study. Please note that your participation is completely 
voluntary and may be discontinued at any time. No penalties or loss of benefits to which you are otherwise entitled shall 
occur should you refuse to participate in the research. If you have questions about Project SCALE you may contact Jack 
Deskins, Coordinator, at the VWWDE Office of Instruction at 304.558.5325 or jdeskins@access.k12.wv.us. Additionally, if 
you have questions about the research associated with the program or this data collection activity, you may contact Pat 
Hammer, Coordinator, at the WDE Office of Research at 304.558.2546 or phammer@access.k12.w.us. 


This research study has been reviewed and approved by the West Virginia Department of Education (VWW/DE) Institutional 
Review Board (IRB). Should you desire additional details about this study’s approval status, you may contact the VWW/DE 
IRB chairperson, Nathaniel Hixson (nhixson@access.k12.wv.us). 


Again, thank you for your participation! 
Section I: About You 


1. Select your school district below. 


[| 


2. Please select your school from the list below. 


[| 


Other (please specify) 
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3. What grade level(s) do you teach (select all that apply)? 


4. Which of the following best describes your primary content area? 
O Arts (Music, Dance, Theater, Visual Art) 


O Physical Education 


O Elementary Education 


O Title | 


Other (please specify) 


Section Il: Your Instruction 


5. In the most recent school year, how often did you integrate any of the following 
strategies in your classroom lessons? 


never rarely 


sometimes 


2 
= 
o 
= 
si 


Music 

Drama/Acting 

Visual Art 

Creative Writing 

Physical Education/Sports 


Dance/Movement 


OOQO00O0 
OQOO0O0O0 
OOQOOO00O0 
OQ0O00O0 
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6. In the most recent school year, how often did you collaborate with the following people 
in designing lessons for your own class? 


Q 
a 
o 


sometimes 


a 
3 
< 


Music Teacher 

Art Teacher 

Physical Education Teacher 
Special Education Teacher 
Title | Teacher 

General Classroom Teachers 
Principal 

Community Members 


Parents 


OOOOCOOO00O: 


O O 
O O 
6 @ 
O O 
O O 
© O 
O O 
O O 
O O 


7. In the most recent school year, how often did you collaborate with the following groups 
to develop lessons for your classes? 


OQOQOOO000O 
OQOQOO0O000OO 


rarely sometimes 


| developed lessons with my grade level peers 
| developed my lessons alone 


| developed lessons with other/all professional staff 


Section III: About the Students in Your School 
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Project SCALE School Survey [WVDE-CIS-94] 


8. Please indicate the extent to which you agree with the following statements about 
students at your school. Please consider the most recent school year as you answer each 
question. 


Students... 


strongly disagree disagree 


My 
a 
a 
o 
o 


strongly agree 
enjoy learning 

believe that school is a fun place to be 

feel close to other people at school 

feel like they belong 

are happy to be at school most days 

like most of their teachers 

feel like they will fail no matter how hard they try 
are not interested in school 


feel that most teachers care about how well they are 
doing in class 


respect most of their teachers 
feel that school is a waste of time 
are excited about their work at school 


feel that school rules are fair 


OOOO OCOOOOO0O0O0O 
OOOO OOOOOO0O0O0O 
OOOO OOOCCOCOCOOO: 
OOOO OOOOOOO0OO 
OOOO OOQOOOO00O0O 


Section III: About the Students in Your School 


9. Please indicate the extent to which you agree with the following statements about 
students at your school. Please consider the most recent school year as you answer each 
question. 


Students... 


strongly disagree disagree 


iy 
a 
2 
@ 
oO 


strongly agree 


stay on task 

complete their homework regularly 

read books outside of school 

follow instructions in class 

get into trouble in my class 

try to find ways to stay home from school 
participate in the arts 


participate in sports 


OCOOCOO0O 
OCOOCOO0O 
OOOOOCO0O0O0: 
OQCOOCOOCO 
OOCOOCOOCO 


participate in other extracurricular activities (e.g., clubs) 
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Section III: About the Students in Your School 


10. Please indicate the extent to which you agree with the following statements about 
students at your school. Please consider the most recent school year as you answer each 
question. 


Students... 


strongly disagree disagree neutral 


My 
Q 
co 
o 
oO 


strongly agree 
appear genuinely interested in class 
consistently pay attention 

are motivated to do well 

prefer challenging assignments 

study at home even when they don't have a test 
come to school prepared to learn 

feel that classes are boring 

try their best on everyday assignments 

try their best on tests 

feel they are getting a good education 

believe they are learning a lot in their classes 


are off task (daydreaming) in class 


OOQOOOOOOCOOO00O 
OOQOOOOOOCOO0O00O 
OOQOOOCOOCOCO0OO00OO 
OOO0O0OCOOOCO0O0O0O 
OOOOOOOOO0000O 


often just pretend they are working in class 


Section IV: Practices in your School 
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Project SCALE School Survey [WVDE-CIS-94] 


11. Please indicate the extent to which you agree with each of the following statements 
about your school. Please consider the most recent school year as you answer each 


question. 


strongly 


: disagree neutral 
disagree 


Faculty are encouraged to exercise initiative for change to improve their performance 
There is collaboration among faculty 

The principal uses professional feedback from teachers 

Professional trust is evident among the faculty 

There are channels for open communication among the school staff 

Those affected by a decision play a significant role in the decision-making process 
Leadership within the school is open to anyone willing to assume responsibility 
Administrators include teachers in the decision-making process 

Faculty have the power to act on their decisions 

Faculty respect each other professionally 

Faculty work together to to seek solutions to problems 


Administrators are team players 


OQOOOOOOOO00O0O 
OOQOQOOOOCO00CO0O 
OOQOOOOO0C0COC0COO 
OOOOOOOCO00C0O 
OQOOOOOOOCOO00O0O 


The principal is receptive to various points of view 


Section IV: Practices in your School 
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12. Please indicate the extent to which you agree with each of the following statements 
about your school. Please consider the most recent school year as you answer each 


question. 


strongly 


. disagree neutral 
disagree 


Students are persistent in completing difficult tasks 


Parents’ behaviors indicate a belief that success in school is dependent on student 
effort 


Students are intrinsically motivated by attempts to improve student learning 
Students exercise control over their own learning 
Students look for ways to improve their own performance 


Parents’ behaviors indicate that they feel their efforts at home do affect their children's 
success in school 


Students take pride in the physical appearance of their school 
The intrinsic motivation of students increases as they move through this school 


Students view assessment as a means to give them feedback on their learning--not 
only as an end in and of itself 


Students accept responsibility for their own performance 
Students are aware of their own learning strengths 


Students believe that hard work pays off 


e ©@ 
ome 
oe @ 
OQ. © 
e @ 
Oo -© 
e @ 
ome 
e @ 
OO -) 
e @ 
O O 


OOO OOO OOOO OO 
OOO OOO OOOO CO 3 
ODORDOD OVOD OO 


Section IV: Practices in your School 
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13. Please indicate the extent to which you agree with each of the following statements 
about your school. Please consider the most recent school year as you answer each 


question. 


strongly 


: disagree neutral 
disagree 


Faculty consistently consider how teaching/learning can be improved 

Teachers are sensitive to different student learning styles 

Students are taught to build on their strongest learning modes 

Collaboration among faculty is motivated by attempts to improve student learning 
Teachers vary their instruction to accommodate different learning styles 


When outcomes are less than desired, faculty increase their efforts to attain unmet 
goals 


Faculty view accountability as a positive concept 
Teachers look for ways to improve their own performance 
Teachers encourage student questioning 

Professional staff value input from students 

Students are encouraged to learn with one another 


Teachers use instructional practices that stimulate curiosity 


QOOOOO00O0 OOQOQO0O0O 
OOO0000 OOOOOO 
OOOO0000 OOOO0O0O 
MOUOUON OOOU0D 
QOQOOOO00O0 OQOOO0O0O 


Faculty perceive the vision as including a shared responsibility for high levels of 
student learning 


Section IV: Practices in your School 


14, Please indicate the extent to which you agree with each of the following statements 
about your school. Please consider the most recent school year as you answer each 


question. 


strongly . strongly 
7 disagree neutral agree 
disagree 


The entrance to the school is welcoming to visitors 
The school gives an appearance of being safe 
The school provides an inviting appearance 


The physical environment of this school is maintained so that the building appears 
clean 


There are signs that help visitors find the locations they are looking for in our building 


Section V: Any Additional Feedback 
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15. Please provide any additional comments you have about Project SCALE. 


4 


The Survey is Complete! 


Thank you for your time. Please click "done" to submit your response. 
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E-mail invitations 
ANNOUNCEMENT 
Dear West Virginia Educator, 


In a few days, you will receive an e-mail invitation, sent via SurveyMonkey, to partic- 
ipate in a survey we are conducting with regard to Project SCALE—a collaboration between 
the West Virginia Symphony Orchestra and the WVDE Office of Instruction. The results of 
this survey will be used to help the Symphony and the Department understand what is going 
well—or not so well—with this innovative project. 


The survey contains a variety of sections and may take up to 20 minutes to complete. 
We urge you to watch for this invitation and to take a few minutes to respond as soon as you 
receive the message. Your honest impressions, whether favorable or unfavorable, will be 
greatly appreciated. 


If you have questions about this survey, you may contact me at the WVDE Office of 
Research at 304.558.2546 or phammer@access.ki2.wv.us. If you have questions about Pro- 
ject SCALE you may contact Jack Deskins, Coordinator at the WVDE Office of Instruction, at 


304.558.5325 or jdeskins@access.k12.wv.us. 


Best regards, 


Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 

Charleston, WV 25305 
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FIRST REQUEST 
Subject Line: Project SCALE—A Survey Request from the WVDE Office of Research 
Dear West Virginia Educator, 


The West Virginia Symphony Orchestra and the WVDE Office of Instruction have 
asked us to study the implementation of Project SCALE at your school and the other partici- 
pating schools. As a part of that study, we would like to learn more about your current in- 
structional practices and your perceptions of everyday activities in your school. The results 
of this survey will be used to help the Symphony and the Department understand what is 
going well—or not so well—with this innovative project. 


The survey contains a variety of sections and may take up to 20 minutes to complete. 
We urge you to respond as soon as you can by visiting the following website: 
https://www.surveymonkey.com/s/ProjectScaleSchoolSurvey. Your honest impressions, 
whether favorable or unfavorable, will be greatly appreciated. 


Your responses to the survey are confidential and will only be reported in the aggre- 
gate. The survey is voluntary, and there are no consequences should you decide not to partic- 
ipate. However, we strongly encourage your participation so that we can learn more about 
your experience with the project. 


If you have questions about this survey, you may contact Pat Hammer, Coordinator, 
at the WVDE Office of Research at 304.558.2546 or phammer@access.ki2.wv.us. If you have 
questions about Project SCALE you may contact Jack Deskins, Coordinator at the WVDE 
Office of Instruction, at 304.558.5325 or jdeskins@access.k12.wv.us. 


Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 

Charleston, WV 25305 


64 | Evaluation of the Student-Centered Arts-Learning Environments (SCALE) Project 


Appendix C. Survey Instruments 


SECOND REQUEST 
Subject Line: Project SCALE—Second Request from the WVDE Office of Research 
Dear West Virginia Educator, 


A few days ago, we contacted you about a survey we are conducting for the West Vir- 
ginia Symphony Orchestra and the WVDE Office of Instruction, to learn about the imple- 
mentation of Project SCALE at your school. We would like to know about your current 
instructional practices and your perceptions of everyday activities in your school, as a part of 
an evaluation study. 


The survey contains a variety of sections and may take up to 20 minutes to complete. 
We urge you to respond as soon as you can by going to the following website: 
https://www.surveymonkey.com/s/ProjectScaleSchoolSurvey. Your honest impressions, 
whether favorable or unfavorable, will be greatly appreciated. 


Your responses to the survey are confidential and will only be reported in the aggre- 
gate. The survey is voluntary, and there are no consequences should you decide not to partic- 
ipate. However, we strongly encourage your participation so that we can learn more about 
your experience with the project. 


If you have questions about this survey, you may me at the WVDE Office of Research 
at 304.558.2546 or phammer@access.ki2.wv.us. If you have questions about Project SCALE 
you may contact Jack Deskins, Coordinator at the WVDE Office of Instruction, at 
304.558.5325 or jdeskins@access.k12.wv.us. 


Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 

Charleston, WV 25305 
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THIRD REQUEST 


Subject Line: Project SCALE—Your Response Urgently Requested from the WVDE Office of 
Research 


Dear West Virginia Educator, 


Once again, we are contacting you about a survey we are conducting for the West 
Virginia Symphony Orchestra and the WVDE Office of Instruction, to learn about the im- 
plementation of Project SCALE at your school. We would like to know about your current 
instructional practices and your perceptions of everyday activities in your school, as a part of 
an evaluation study. 


The survey contains a variety of sections and may take up to 20 minutes to complete. 
We urge you to respond as soon as you can by visiting the following website: 
https://www.surveymonkey.com/s/ProjectScaleSchoolSurvey. Your honest impressions, 
whether favorable or unfavorable, will be greatly appreciated. 


Your responses to the survey are confidential and will only be reported in the aggre- 
gate. The survey is voluntary, and there are no consequences should you decide not to partic- 
ipate. However, we strongly encourage your participation so that we can learn more about 
your experience with the project. 


If you have questions about this survey, you may contact Pat Hammer, Coordinator, 
at the WVDE Office of Research at 304.558.2546 or phammer@access.ki2.wv.us. If you have 
questions about Project SCALE you may contact Jack Deskins, Coordinator at the WVDE 
Office of Instruction, at 304.558.5325 or jdeskins@access.k12.wv.us. 


Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 

Charleston, WV 25305 
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FOURTH REQUEST 
Subject Line: Project SCALE—Please respond! WVDE Office of Research 
Dear West Virginia Educator, 


Once again, we are contacting you about a survey we are conducting for the West 
Virginia Symphony Orchestra and the WVDE Office of Instruction, to learn about the im- 
plementation of Project SCALE at your school. We would like to know about your current 
instructional practices and your perceptions of everyday activities in your school, as a part of 
an evaluation study. 


The survey contains a variety of sections and may take up to 20 minutes to complete. 
We urge you to respond as soon as you can by visiting the following website: 
https://www.surveymonkey.com/s/ProjectScaleSchoolSurvey. Your honest impressions, 
whether favorable or unfavorable, will be greatly appreciated. 


Your responses to the survey are confidential and will only be reported in the aggre- 
gate. The survey is voluntary, and there are no consequences should you decide not to partic- 
ipate. However, we strongly encourage your participation so that we can learn more about 
your experience with the project. 


If you have questions about this survey, you may contact Pat Hammer, Coordinator, 
at the WVDE Office of Research at 304.558.2546 or phammer@access.ki2.wv.us. If you have 
questions about Project SCALE you may contact Jack Deskins, Coordinator at the WVDE 
Office of Instruction, at 304.558.5325 or jdeskins@access.k12.wv.us. 


Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 

Charleston, WV 25305 
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Project SCALE Post-Intervention School Survey (Post Survey) 


Survey instrument 


ad Ke) (=Xe4Cc-48] of] os) OF VM mk 40] 0)-] OM axes) coal (AY =) ani(e) amosergcele) mole lay/-\Vm AUAYAVA D) mts 


Welcome to the Project SCALE Post-Intervention School Survey 


Welcome and thank you for agreeing to take part in this important data collection activity. 


By participating in this data collection activity, you are consenting to participate in a research study, the purpose of 
which is to determine the effectiveness of an educational program. The results of this research study will also be used to 
help determine revisions to the program and possibly to inform the development of future education programs. Your 
participation in this study shall be limited to completing a brief survey and should not take more than 20 minutes of your 
time. You will be presented with a series of items and asked to indicate your responses by either providing a rating or a 
short open-ended response. 


Participation in this research study poses no more risk than that which you would encounter during the course of a 
normal day. While you may not experience tangible direct benefits from participating in this research study, indirect 
benefits may include the program being better tailored to meet your needs and/or improved outcomes for study 
participants and/or students. All responses to this data collection activity shall remain completely confidential and no 
identification of individual participants will occur. All results shall be reported in aggregate. 


There is no compensation for taking part in this research study, although you do have an approximate one in 50 chance 
of winning a Best Buy gift certificate valued at $50 to $250 and two West Virginia Symphony Orchestra concert tickets. 
Please note that your participation is completely voluntary and may be discontinued at any time. No penalties or loss of 
benefits to which you are otherwise entitled shall occur should you refuse to participate in the research. If you have 
questions about Project SCALE you may contact Jack Deskins, Coordinator, at the WVDE Office of Instruction at 
304.558.5325 or jdeskins@access.k12.wv.us. Additionally, if you have questions about the research associated with the 
program or this data collection activity, you may contact Pat Hammer, Coordinator, at the VWW/DE Office of Research at 
304.558.2546 or phammer@access. k12.w.us. 


This research study has been reviewed and approved by the West Virginia Department of Education (VWW/DE) Institutional 
Review Board (IRB). Should you desire additional details about this study's approval status, you may contact the WDE 
IRB chairperson, Nathaniel Hixson (nhixson@access.k12.wv.us). 


Again, thank you for your participation! 
Section I: About You 


1. Select your school district below. 


2. Please select your school from the list below. 


[ 


Other (please specify) 
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Project&nbsp;SCALE&nbsp;Post-Intervention School Survey [WVDE- 
3. What grade level(s) do you teach (select all that apply)? 


4. Which of the following best describes your primary content area? 


O Arts (Music, Dance, Theater, Visual Art) 


O Physical Education 


O Elementary Education 


O Title | 


Other (please specify) 


Section Il: Your Instruction 


5. This school year, how often have you integrated any of the following strategies in your 
classroom lessons? 


=] 
oO 
a 
s. 
“< 


sometimes 


g 
o 
= 
sj 


Music 

Drama/Acting 

Visual Art 

Creative Writing 

Physical Education/Sports 


Dance/Movement 


OOQOO0O0O0 
OOQOO00O0 
QOQOOO0O0O0 
OQO0O0O0 
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6. This school year, how often have you collaborated with the following people in 
designing lessons for your own class? 


fe) 
= 
o 
= 
a 


sometimes 


3 
g 
g 
a 

< 


Music Teacher 

Art Teacher 

Physical Education Teacher 
Special Education Teacher 
Title | Teacher 

General Classroom Teachers 
Principal 

Community Members 


Parents 


OOOO0O0CO00O 
OOOO0O0C00O 
OOOO0O000O 
OOOO0O000O 
OQOOO0O000O 


7. This school year, how often have you collaborated with the following groups to develop 
lessons for your classes? 


rarely sometimes 


| developed lessons with my grade level peers 
| developed my lessons alone 


| developed lessons with other/all professional staff 


Section III: About the Students in Your School 
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8. Please indicate the extent to which you agree with the following statements about 
students at your school this school year. 


Students... 


strongly disagree disagree 


My 
a 
a 
o 
o 


strongly agree 


enjoy learning 

believe that school is a fun place to be 

feel close to other people at school 

feel like they belong 

are happy to be at school most days 

like most of their teachers 

feel like they will fail no matter how hard they try 
are not interested in school 


feel that most teachers care about how well they are 
doing in class 


respect most of their teachers 
feel that school is a waste of time 


are excited about their work at school 


OCOO OOCCOOCOO 

OCOO OOCOOCOOO 
OOOO OOOOCOCCOCOO: 
OCOO OOCOOCOOO 
OCOO OOCOOCOOO 


feel that school rules are fair 


Section III: About the Students in Your School 


9. Please indicate the extent to which you agree with the following statements about 
students at your school this school year. 


Students... 


strongly disagree disagree 


0 
a 
Ss 
i) 
oO 


strongly agree 


stay on task 


complete their homework regularly 


OO 


read books outside of school 

follow instructions in class 

get into trouble in my class 

try to find ways to stay home from school 
participate in the arts 


participate in sports 


OCOOCOOCO 
OCOOCOO 
OOOCCCOCOCO: 
OCOOCO0CO 
OCOOCO0CO 


participate in other extracurricular activities (e.g., clubs) 


Section III: About the Students in Your School 
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10. Please indicate the extent to which you agree with the following statements about 
students at your school this school year. 


Students... 


strongly disagree disagree 


og 
a 
= 
i) 
i) 


strongly agree 


appear genuinely interested in class 
consistently pay attention 

are motivated to do well 

prefer challenging assignments 

study at home even when they don't have a test 
come to school prepared to learn 

feel that classes are boring 

try their best on everyday assignments 

try their best on tests 

feel they are getting a good education 
believe they are learning a lot in their classes 
are off task (daydreaming) in class 


often just pretend they are working in class 


O O 
O O 
O O 
© O 
O O 
O O 
O O 
O O 
O O 
O O 
O O 
O O 
O O 


OOO0OO0O0O0CO0COCOO 
OQOO0O0O0O0OCOCCOCO0CO 
OQOQOOOOOO00CO0O0O 


Section IV: Practices in your School 


11. Please indicate the extent to which you agree with each of the following statements 


about your school this school year. 


strongly 


A disagree neutral 
disagree 


Faculty are encouraged to exercise initiative for change to improve their performance 
There is collaboration among faculty 

The principal uses professional feedback from teachers 

Professional trust is evident among the faculty 

There are channels for open communication among the school staff 

Those affected by a decision play a significant role in the decision-making process 
Leadership within the school is open to anyone willing to assume responsibility 
Administrators include teachers in the decision-making process 

Faculty have the power to act on their decisions 

Faculty respect each other professionally 

Faculty work together to to seek solutions to problems 


Administrators are team players 


OQOOOOOO0CO00OO 
OQOQOO0O0OO00C0C0C0O 
OOOOOOOCOCOC0COCO 
OQOOOOOO0CO000O0O 
OQOOOOOOOO000O 


The principal is receptive to various points of view 


Section IV: Practices in your School 
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12. Please indicate the extent to which you agree with each of the following statements 


about your school this school year. 


strongly 


. disagree neutral 
disagree 


My 
a 
= 
o 
(3) 


Students are persistent in completing difficult tasks 


Parents' behaviors indicate a belief that success in school is dependent on student 
effort 


Students are intrinsically motivated by attempts to improve student learning 
Students exercise control over their own learning 
Students look for ways to improve their own performance 


Parents’ behaviors indicate that they feel their efforts at home do affect their children's 
success in school 


Students take pride in the physical appearance of their school 
The intrinsic motivation of students increases as they move through this school 


Students view assessment as a means to give them feedback on their learning--not 
only as an end in and of itself 


Students accept responsibility for their own performance 


Students are aware of their own learning strengths 


OOO OOO OOOO OO 
OOO OOO OOOO OO 
OOO OOO OOOO OO 
OOO OOO OOOO OO 
OOO OOO OOQOVO OO 


Students believe that hard work pays off 


Section IV: Practices in your School 
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13. Please indicate the extent to which you agree with each of the following statements 


about your school this school year. 


strongly 


fi disagree neutral 
disagree 


Faculty consistently consider how teaching/learning can be improved 

Teachers are sensitive to different student learning styles 

Students are taught to build on their strongest learning modes 

Collaboration among faculty is motivated by attempts to improve student learning 
Teachers vary their instruction to accommodate different learning styles 


When outcomes are less than desired, faculty increase their efforts to attain unmet 
goals 


Faculty view accountability as a positive concept 
Teachers look for ways to improve their own performance 
Teachers encourage student questioning 

Professional staff value input from students 

Students are encouraged to learn with one another 


Teachers use instructional practices that stimulate curiosity 


OOOO00ON OOQOOOOO 
OOOO000CO0 OOOOOO 
OOOO00ON OOOOOO 
OOOO00ON OOOOOO 
OOOOO00ON OOOOO0O 


Faculty perceive the vision as including a shared responsibility for high levels of 
student learning 


Section IV: Practices in your School 


14, Please indicate the extent to which you agree with each of the following statements 


about your school this school year. 


strongly 


. disagree neutral agree 
disagree 


The entrance to the school is welcoming to visitors 
The school gives an appearance of being safe 
The school provides an inviting appearance 


The physical environment of this school is maintained so that the building appears 
clean 


There are signs that help visitors find the locations they are looking for in our building 
Section V: Any Additional Feedback 


15. Please provide any additional comments you have about Project SCALE. 


~ 


The Survey is Complete! 
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Thank you for your time. Please click “done” to submit your response. 
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E-mail invitations 
ANNOUNCEMENT 
Dear Project SCALE Participant, 


Thank you again for responding last fall to our survey about Project SCALE—a col- 
laboration between the West Virginia Symphony Orchestra (WVSO) and the WVDE Office of 
Instruction. In a few days, you will receive an e-mail invitation, sent via SurveyMonkey, to 
participate in a follow-up survey we are conducting to learn about what went well—or not so 
well—with this project. 


Only you and other individuals who responded to the first survey are eligible 
to participate in this follow-up survey and have the chance to win one of the following 
prizes provided by the Symphony: 


First prize—a $250 Best Buy gift certificate 
Second prize—a $100 Best Buy gift certificate 
Third prize—a $50 Best Buy gift certificate 


The prizes are for personal use, and each is accompanied by a gift certificate for 
two WVSO concert tickets. Winners will be selected in a random drawing of follow-up sur- 
vey respondents. 


The survey contains a variety of sections and may take up to 20 minutes to com- 
plete. We urge you to watch for this invitation and to take a few minutes to respond as soon 
as you receive the message. Your honest impressions, whether favorable or unfavorable, 
will be greatly appreciated. 


If you have questions about this survey, you may contact me at the WVDE Office of 
Research at 304.558.2546 or phammer@access.k12.wv.us. If you have questions about 
Project SCALE you may contact Jack Deskins, Coordinator at the WVDE Office of Instruc- 


tion, at 304.558.5325 or jdeskins@access.k12.wv.us. 


Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 
Charleston, WV 25305 
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FIRST REQUEST 
Subject Line: Project SCALE—A Survey Request from the WVDE Office of Research 
Dear Project SCALE Participant, 


The West Virginia Symphony Orchestra and the WVDE Office of Instruction have asked us to 
study the implementation of Project SCALE at your school and the other participating schools. As a 
part of that study, we would like to learn more about your current instructional practices and your per- 
ceptions of everyday activities in your school. The results of this survey will be used to help the Sym- 
phony and the Department understand what is going well—or not so well—with this innovative 
project. 


The survey contains a variety of sections and may take up to 20 minutes to complete. We 
urge you to respond as soon as you can by visiting the following website: 


https ://www.surveymonkey.com/s/ProjectScaleSchoolSurvey. Your honest impressions, whether fa- 
vorable or unfavorable, will be greatly appreciated. 


Only you and other individuals who responded to the first survey (last fall) are eligible 
to participate in this follow-up survey and have the chance to win one of the following prizes 
provided by the Symphony: 


First prize—a $250 Best Buy gift certificate 
Second prize—a $100 Best Buy gift certificate 
Third prize—a $50 Best Buy gift certificate 


The prizes are for personal use, and each is accompanied by a gift certificate for two 
WVSO concert tickets. Winners will be selected in a random drawing of follow-up survey respond- 
ents. To qualify, be sure to type in the following Respondent ID where it is called for in the sur- 
vey form: [RID]. This ID is used only so we can identify you if you win one of the prizes. 


Your responses to the survey are confidential and will only be reported in the aggregate. The 
survey is voluntary, and there are no consequences should you decide not to participate. However, 
we strongly encourage your participation so that we can learn more about your experience with the 
project. 


If you have questions about this survey, you may contact Pat Hammer, Coordinator, at the 
WVDE Office of Research at 304.558.2546 or phammer@access.k12.wv.us. If you have questions 
about Project SCALE you may contact Jack Deskins, Coordinator at the WVDE Office of Instruction, 


at 304.558.5325 or jdeskins@access.k12.wv.us. 
Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 
Charleston, WV 25305 
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SECOND REQUEST 
Subject Line: Project SCALE—Second Request from the WVDE Office of Research 
Dear Project SCALE Participant, 


A few days ago, we contacted you about a survey we are conducting for the West Virginia 
Symphony Orchestra and the WVDE Office of Instruction, to learn about the implementation of Pro- 
ject SCALE at your school. We would like to know about your current instructional practices and your 
perceptions of everyday activities in your school, as a part of an evaluation study. 


The survey contains a variety of sections and may take up to 20 minutes to complete. We 
urge you to respond as soon as you can by going to the following website: [SurveyLink]. Your honest 
impressions, whether favorable or unfavorable, will be greatly appreciated. 


Only you and other individuals who responded to the first survey (last fall) are eligible 
to participate in this follow-up survey and have the chance to win one of the following prizes 
provided by the Symphony: 


First prize—a $250 Best Buy gift certificate 
Second prize—a $100 Best Buy gift certificate 
Third prize—a $50 Best Buy gift certificate 


The prizes are for personal use, and each is accompanied by a gift certificate for two 
WVSO concert tickets. Winners will be selected in a random drawing of follow-up survey respond- 
ents. To qualify, be sure to type in the following Respondent ID where it is called for in the sur- 
vey form: [CustomData]. This ID is used only so we can identify you if you win one of the prizes. 


Your responses to the survey are confidential and will be reported only in the aggregate. The 
survey is voluntary, and there are no consequences should you decide not to participate. However, 
we strongly encourage your participation so that we can learn more about your experience with the 
project. 


If you have questions about this survey, you may me at the WVDE Office of Research at 
304.558.2546 or phammer@access.k12.wv.us. If you have questions about Project SCALE you may 
contact Jack Deskins, Coordinator at the WVDE Office of Instruction, at 304.558.5325 or jde- 
skins@access.k12.Wv.us. 


Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 
Charleston, WV 25305 
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THIRD REQUEST 


Subject Line: Project SCALE—Your Response Urgently Requested from the WVDE Office of Re- 
search 


Dear Project SCALE Participant, 


Once again, we are contacting you about a survey we are conducting for the West Virginia 
Symphony Orchestra and the WVDE Office of Instruction, to learn about the implementation of Pro- 
ject SCALE at your school. We would like to know about your current instructional practices and your 
perceptions of everyday activities in your school, as a part of an evaluation study. 


The survey contains a variety of sections and may take up to 20 minutes to complete. We 
urge you to respond as soon as you can by visiting the following website: [SurveyLink]. Your honest 
impressions, whether favorable or unfavorable, will be greatly appreciated. 


Only you and other individuals who responded to the first survey (last fall) are eligible 
to participate in this follow-up survey and have the chance to win one of the following prizes 
provided by the Symphony: 


First prize—a $250 Best Buy gift certificate 
Second prize—a $100 Best Buy gift certificate 
Third prize—a $50 Best Buy gift certificate 


The prizes are for personal use, and each is accompanied by a gift certificate for two 
WVSO concert tickets. Winners will be selected in a random drawing of follow-up survey respond- 
ents. To qualify, be sure to type in the following Respondent ID where it is called for in the sur- 
vey form: [CustomData]. This ID is used only to allow us to identify you if you win one of the prizes. 


Your responses to the survey are confidential and will only be reported in the aggregate. The 
survey is voluntary, and there are no consequences should you decide not to participate. However, 
we strongly encourage your participation so that we can learn more about your experience with the 
project. 


If you have questions about this survey, you may contact me at 304.558.2546 or pham- 
mer@access.k12.wv.us. If you have questions about Project SCALE you may contact Jack Deskins, 
Coordinator at the WVDE Office of Instruction, at 304.558.5325 or jdeskins@access.k12.wv.us. 


Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 
Charleston, WV 25305 
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FOURTH REQUEST 
Subject Line: Project SCALE—There is still time! WVDE Office of Research 
Dear Project SCALE Participant, 


Once again, we are contacting you about a follow-up survey we are conducting for the West 
Virginia Symphony Orchestra and the WVDE Office of Instruction, to learn about the implementation 
of Project SCALE at your school. We would like to know about your current instructional practices and 
your perceptions of everyday activities in your school, as a part of an evaluation study. 


The survey contains a variety of sections and may take up to 20 minutes to complete. We 
urge you to respond as soon as you can by visiting the following website: [SurveyLink]. Your honest 
impressions, whether favorable or unfavorable, will be greatly appreciated. 


Only you and other individuals who responded to the first survey (last fall) are eligible 
to participate in this follow-up survey and have the chance to win one of the following prizes 
provided by the Symphony: 


First prize—a $250 Best Buy gift certificate 
Second prize—a $100 Best Buy gift certificate 
Third prize—a $50 Best Buy gift certificate 


The prizes are for personal use, and each is accompanied by a gift certificate for two 
WVSO concert tickets. Winners will be selected in a random drawing of follow-up survey respond- 
ents. To qualify, be sure to type in the following Respondent ID where it is called for in the sur- 
vey form: [CustomData]. This ID is used only to allow us to identify you if you win one of the prizes. 


Your responses to the survey are confidential and will only be reported in the aggregate. The 
survey is voluntary, and there are no consequences should you decide not to participate. However, 
we strongly encourage your participation so that we can learn more about your experience with the 
project. 


If you have questions about this survey, you may contact Pat Hammer, Coordinator, at the 
WVDE Office of Research at 304.558.2546 or phammer@access.k12.wv.us. If you have questions 
about Project SCALE you may contact Jack Deskins, Coordinator at the WVDE Office of Instruction, 


at 304.558.5325 or jdeskins@access.k12.wv.us. 
Best regards, 
Pat Hammer 


Patricia Cahape Hammer 

Coordinator 

Office of Research 

West Virginia Department of Education 
Building 6, Room 722 

1900 Kanawha Boulevard East 
Charleston, WV 25305 
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Table A1. Status of SCALE Schools: Title |, Federal NCLB School Improvement Requirements 


NCLB school improvement SCALE project 

Title | requirements SIG involvement 
SCALE project school County (SW) SC SES CA’ R1~ R2_~— school New Previous 
Ansted Elementary Fayette Y Y Y 
Brookview Boone Y Y Y Y 
Burch Elementary Mingo Y Y 
Culloden Elementary Cabell Y 
Dingess Elementary Mingo Y Vf 
Doddridge County Elementary Doddridge Y Y Y Y Y Y Y Y 
Geary Elementary/Middle Roane Y Y Y Y Y 
Guyandotte Elementary Cabell Y Y Y 
Lizemore Elementary Clay Y Y 
Poca Elementary Putnam Y Y 
Reedy Elementary Roane Y Y 
Romney Elementary Hampshire Y Y Y Y Y Y 
Smoot Elementary Greenbrier Y Y 
Spencer Elementary Roane Y Y Y Y Y 
Watts Elementary Kanawha VW Y Y Y Y Y 
Weimer Elementary Kanawha Y Y 


SW = school-wide, SC = school choice, SES = supplemental educational services, CA = corrective action, R1 = 
restructuring plan for alternative governance, R2 = restructuring implement alternative governance, SIG = 
School Improvement Grant participant 

Information in this table based primarily on the following West Virginia Department of Education online re- 
sources: 2012-2013 Title | Schools (retrieved from http://wvde.state.wv.us/titlei/titlei_schools.html)and 

Title | 2012-2013 Sanctions: Identified for Title | School Improvement (retrieved from http://wvde.state.wv.us/ 
titlei/lea_timeline.html). Other information was supplied by the WVSO. 
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Table A 2. Quality, Relevance, and Usefulness of SCALE Project Professional Development 


n Percent 


Which of the following statements best describes the usefulness of the PD you received as part of Project 
SCALE? 


Total 28 100.0 
It was a good start. 6 21.4 
It was a good start, but | have a lot of questions. 1 3.6 
It was a good start, and | look forward to using what | learned in my classroom (or work setting). 16 57.1 
It provided everything | need to use what | learned in my classroom (or work setting). 5 17.9 
| don’t think these ideas will work in my classroom (or work setting). 0 0 
It’s too early to tell. 6) 0 
Indicate the extent to which the PD you received as part of Project SCALE met your professional needs. 

Total 28 100.0 
It addressed my professional learning needs completely. 7 25.0 
It addressed some of my professional learning needs. 21 75.0 
It did not address my professional learning needs. 0 0 
This professional development did not help much because | already know what | need to know 0 0 


about this topic. 
Which of the following statements best describes the likelihood that you will apply what you learned in this PD 
in your classroom (or work setting)? 

Total 28 100.0 
| already practice/apply the knowledge/skills this training this training provided in my classroom 13. 46.4 

(or work setting), and it seems to work well. 


| have already practiced/applied the knowledge/skills this training provided in my classroom (or 0 0 
work setting), but it is not appropriate for my students. 

| look forward to practicing/applying the knowledge/skills in my classroom (or work setting) 13 46.4 
during the upcoming school year. 

| don’t think what | learned here will work for my students so | don’t envision applying the 0 0 
knowledge/skills. 

No response 2 7.1 


To what extent was the PD/training aligned with your school’s/program’s goals for improving instruction? 
Total 28 100.0 
The PD was VERY CLOSELY aligned with school’s/program’s goal for instructional improvement. 16 57.1 


The PD was SOMEWHAT aligned with school’s/program’s goals for instructional improvement. 10 35.7 
The PD was NOT ALIGNED with school’s/program’s goals for instructional improvement. 2 7.1 
The PD was INCONSISTENT with school’s/program’s goals for instructional improvement. 0 0 
| don’t know. 0 0 


Which of the following statements best describes how the Project SCALE PD opportunity you attended 
compares with other PD opportunities in which you have participated in the last three years? 
Total 28 100.0 
This professional development was MORE USEFUL than other professional development | have 18 64.3 
participated in. 
This professional development was ABOUT THE SAME AS other professional development | 8 28.6 
have participated in. 


This professional development was less USEFUL than other professional development | have 0 0 
participated in. 

| don’t have an opinion. 2 7.1 

| haven’t participated in any other professional development in the last three years. 0 
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Table A 3. Ability of Students to Engage in Supported Behaviors as a Result of SCALE Project, by School 
Collab- — Partici- 
Connect orate patein 
content Share Partici- meaning —higher- Work 
among’ Repre- created pate -fully order Engage _ collab- 
multiple sent products fully as with learning meaning oratively 
discip- complex with _ indivi- each conver- -fullyin with the Mean 

School lines ideas others duals other sations thearts WVSO score 
Mean Scores 3.4 3.2 3.5 3.5 3.3 3.1 3.5 3.1 
Doddridge 4 4 4 4 4 4 4 4 4.0 
Guyandotte 4 4 4 4 4 4 4 4 4.0 
Poca 4 4 4 4 4 4 4 4 4.0 
Spencer 4 4 4 4 4 4 4 4 4.0 
Burch 4 4 4 4 4 4 4 3 3.9 
Dingess 4 4 4 4 4 3 4 4 3.9 
Weimer 4 4 4 4 4 3 4 4 3.9 
Culloden 4 4 3 4 3 3 4 4 3.6 
Lizemore 4 3 4 4 4 3 3 3 3.5 
Romney 4 3 4 4 3 3 4 3 3.5 
Brookview 3 3 4 4 3 3 4 1 3.1 
Ansted 3 3 3 3 3 2 3 3 2.9 
Geary 2 2 3 3 3 3 4 3 2.9 
Reedy 3 3 4 3 3 2 3 2 2.9 
Watts 3 2 3 3 3 2 3 3 2.8 


NOTE: Numbers represent level of agreement that students were able to engage in the behaviors described, 


using the following scale: 1 (strongly disagree), 2 (disagree), 3 (agree), or 4 (strongly agree). 
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Table A 4. Groups Involved in the SCALE Project by School 


Percent 
Music Art PhysEd Title! Gen Ed Spec Ed Com- by 
School teacher teacher teacher teacher teacher teacher Admins munity Parents WVSO _ school 
Percent by 
role group 86.7 80.0 53.3 73.3, 100.0 86.7 93.3 20.0 38.5 80.0 
Poca Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 100.0 
Weimer Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 100.0 
Burch Yes Yes Yes Yes Yes Yes Yes No Yes Yes 90.0 
Ansted Yes Yes Yes Yes Yes Yes Yes Yes * No 88.9 
Culloden Yes Yes Yes No Yes Yes Yes No Yes Yes 80.0 
Doddridge Yes Yes Yes Yes Yes Yes Yes No No Yes 80.0 
Geary Yes Yes Yes Yes Yes Yes Yes No No Yes 80.0 
Spencer Yes Yes Yes Yes Yes Yes Yes No No Yes 80.0 
Guyandotte Yes Yes No Yes Yes Yes Yes No No Yes 70.0 
Romney Yes Yes No Yes Yes Yes Yes No No Yes 70.0 
Dingess Yes Yes No No Yes Yes Yes No - Yes 66.7 
Lizemore No No No Yes Yes Yes Yes No Yes Yes 60.0 
Brookview Yes No No Yes Yes No Yes No No No 40.0 
Reedy No No No No Yes Yes Yes No No Yes 40.0 
Watts Yes Yes No No Yes No No No No No 30.0 
Table A5. Arts Content Areas Integrated in the Scale Project by School 
Dance/ Creative Percent by 
School Other movement Music Drama Visual art writing school 
Percent by 13.3 85.7 100.0 100.0 100.0 100.0 
content area 
Doddridge Yes Yes Yes Yes Yes Yes 100.0 
Spencer Yes Yes Yes Yes Yes Yes 100.0 
Ansted No Yes Yes Yes Yes Yes 83.3 
Culloden No Yes Yes Yes Yes Yes 83.3 
Dingess No Yes Yes Yes Yes Yes 83.3 
Guyandotte No Yes Yes Yes Yes Yes 83.3 
Lizemore No Yes Yes Yes Yes Yes 83.3 
Poca No Yes Yes Yes Yes Yes 83.3 
Romney No Yes Yes Yes Yes Yes 83.3 
Watts No Yes Yes Yes Yes Yes 83.3 
Weimer No Yes Yes Yes Yes Yes 83.3 
Brookview No Yes Yes Yes Yes = 80.0 
Burch No ” Yes Yes Yes Yes 80.0 
Geary No No Yes Yes Yes Yes 66.7 
Reedy No No Yes Yes Yes Yes 66.7 
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Table A6. Project SCALE Component Implemented by School 
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Percent by 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 88.9 88.9 77.8 44.4 33.3 22.2 

school 

Team leader 100.0 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y 

identified 

Team identified 80.0 Y Y Y Y Y Y Y Y Y Y Y Y N N N 

Team includes 78.6 Y Y Y * Y Y Y Y Y Y Y Y N N N 

different 

content area 

Team includes 80.0 Y Y Y Y Y Y Y Y Y Y N Y N Y N 

arts teachers 

Team met 64.3 Y Y Y * Y Y Y Y Y N Y N N N N 

regularly 

Arts integration 78.6 Y Y Y i Y Y Y Y Y Y Y Y N N N 

PD offered for 

all staff 

All classes 86.7 Y Y Y Y Y Y Y Y Y Y Y N Y N Y 

participated 

Timely 86.7 Y Y Y Y Y Y Y Y Y Y Y Y Y N N 

communication 

about school 

project 

Support 93.3 Y Y Y Y Y Y Y Y Y Y Y Y Y Y N 


materials shared 
with all 
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Appendix E. Descriptions of SCALE School Projects 


Poca Elementary Implementation Score, 98% 


Project Description: The project was announced at a whole school assembly, inviting students to be as 
creative as they liked. They were provided a number of resources to do so. The primary grades read 
aloud with Reader's Theater, created cutouts of costumes they created, and took Little Red around the 
world to every continent and discussed the weather and transportation there. They also covered 
stranger danger. Other students compared and contrasted Red with other fairy tale characters. Third 
grade students created a puppet theater and acted various scenes from Red. Students wrote across the 
curriculum. They created a grocery list for Red's basket, budgeted the amounts, and visited a grocery 
to purchase the items. Fifth grade put Little Red on trial, each student choosing a part, and chose 
managers for group, such as manager of actors. They also choreographed and performed a dance for 
the play. One class compared the skeleton of the wolf to that of a dog and studied the habitat of the 
wolf. Students created a quilt piece depicting a picture of the story. 


Weimer Elementary Implementation Score, 96.1% 


Project Description: Every teacher and student in each grade, K-5, participated in creating and 
delivering arts integrated activities across content areas. A team of three teachers met regularly, with 
the support of the principal, to direct and support the school-wide activities. Content areas included 
mathematics, language arts, history (an election for President of the Woods, art, music, and 
performing arts. Each classroom contributed to a school mural depicting the Little Red Riding Hood 
story and students from each classroom participated in a culminating celebration, performed for 
parents and guests, on November 9, 2012. 


Spencer Elementary Implementation Score, 92.2% 


Project Description: Third and fourth graders had previously participated in Rumpelstiltzkin with the 
WVSO, so getting whole school involved for Little Red was an easy sell. Some teachers did activities 
for two weeks; some for three or four weeks. The counselor was also involved in addition to related 
arts teachers. There were vertical connections made as 4th graders worked with kindergartners. As 
students entered the Clay Center on concert day, they were so impressed with the building, the 
sculpture, and their favorite word was "AWESOME!" Focus was on integrating the Arts to ELA. Music 
teacher did activities that carried back to ELA. Art teacher was other team member that attended 
SCALE training. PE was heavily involved with movement activities. 


Doddridge Elementary Implementation Score, 91.6% 


Project Description: All grade levels PK through Grade 5 participated; Both horizontal and vertical 
teaming took place; lots of collaboration between general classroom and special education team. Big 
Bad Wolf captured the imagination of students. Teachers found the subject matter to be very 
interactive. Students owned much of the project and will own even more next year. Lots of 
engagement by students and teachers! SCALE project was easily a great followup to school project 
called Camp Yes I Can. Teachers were able to incorporate elements into SCALE and take them farther. 
SCALE project "freed" teachers to collaborate. Assistant Principal Amy Spurlock was team leader who 
attended SCALE trainings in Charleston with one other team member. Title 1 and special education 
teachers were especially pleased to be involved. They felt like many "barriers" were broken down. 
Across the board, there was an overwhelming sense of student pride in artifacts they created. Also a 
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SIG school, teachers overwhelmingly thought this project was great and by the end, not "just one 
more thing on their plates." 


Guyandotte Elementary Implementation Score, 88.2% 


Project Description: 80% of school is on free and reduced lunch. Most of these students would never 
get to go on a trip to hear the symphony. Family support is lacking, and the community is hungry for 
these kinds of experiences. Music teacher was totally invested in this experience, and the teams 
members who attended training managed the PD and gave purpose and enthusiasm to the entire staff. 


Burch Elementary Implementation Score, 87.8% 


Project Description: The school's project was designed to integrate the arts in multiple content areas 
and improve student engagement. The project targeted ONLY the 3rd grade. 


Culloden Elementary Implementation Score, 86.3% 


Project Description: Each classroom, pre-school through grade 5, particpated in the project in a 
variety of ways. Some painted the Red characters and then cut them out and made puppets to reenact 
the play with their classmates. Others incorporated Language Arts and math by comparing versions of 
Red and creating Venn diagrams. Third and fourth graders danced the "Cupid Shuffle" to help them 
remember the rules of rounding. Fifth grade wrote and performed a play based on Red. Art teachers 
incorporated science, discussing trees and insects of the woods and created artworks based on the 
theme. PE teachers taught foreshadowing using music from the provided CD for dance, walking and 
pacing. Music teachers reviewed high and low sounds made by the Red characters and acted out the 
play by using high and low voices. 


Dingess Elementary Implementation Score, 85.9% 


Project Description: Music Teacher Alan Rifle led the charge for Dingess Elementary's project. 
Working closely with team members Barbara Baisden and Andrea Brinegar, the group built upon 
lessons learned in previous year's SCALE project. In first grade, students wrote stories individually 
which teachers then took and morphed into bigger drama productions using as many elements from 
the individual stories as possible. Students worked in groups to mesh/mash their story elements 
together. Using regular events such as Halloween, when costumes would already be worn, students 
performed their plays for other classes. Fourth graders created a math forest in a vacant classroom 
which was left up for a month after its creation. Students wrote reflectively about the creative process; 
teachers documented the process with photographs. When Maestro Cooper visited in December, 
fourth graders guided Maestro Cooper through the math forest, taking pride in describing all the 
measuring they had to do to create the forest. 


Romney Elementary Implementation Score, 83.3% 


Project Description: Project was spearheaded by music and art teachers who got everyone else on 
board. All grades and subjects participated. Further encouragement was given by Transformation 
Specialist who works with the school. 


Lizemore Elementary Implementation Score, 83.2% 


Project Description: All grades were included in Lizemore's SCALE project. Their Title I coordinator 
took the lead in planning but involved all the classroom and special education teachers. There were 
major visual art and theatre components (every class did an original play), as well as some music and 
dance as well. Their primary goals were focused around student engagement. 
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Ansted Elementary Implementation Score, 82.4% 


Project Description: Teachers worked to integrate all subjects into the work surrounding the arts and 
the story of Little Red Riding Hood. Evidence of student work was displayed throughout the school. 
Teachers wanted to expose students to rich learning opportunities in order to engage their population 
in learning. It was expressed that their county leadership requires a focus on skills to the exclusion of 
other subjects. 


Geary Elementary Implementation Score, 73.7% 


Project Description: The project's main goals were increased student engagement, as well as deeper 
understandings in the arts and other subjects. 


Reedy Elementary Implementation Score, 58.6% 


Project Description: The main goal of the school's project was improving student engagement 
through the arts. The project was planned by the entire staff, though one teacher and the principal 
took the lead. All content areas and all grades were included, with some arts integration throughout. 


Brookview Elementary Implementation Score, 53.7% 


Project Description: Music teacher and one other staff member attended the summer training. Music 
teacher was first year teacher and new to the school. Music teacher, Title 1 and general classroom 
teachers in grades 2, 4, 5 were involved. 


Watts Elementary Implementation Score, 50.8% 


Project Description: The school's goals were to enhance the literacy experience of their students by 
exposing them to rich literature and creating concepts which would expand their knowledge and 
vocabulary. Some classrooms used drama and the visual arts. Others used music. The music teacher 
worked with the symphonic music to prepare students. Collectively, students read and composed 
writing in response to Little Red Riding Hood. Some classes used mathematics to further 
understanding. 
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